How the used gear industry is winning the fight against counterfeiters
- 20 June, 2008 10:46
Over 100 attendees gathered for Network World's live chat on counterfeit network gear -- how to detect it and protect yourself -- with guest Mike Sheldon, chief executive officer of Network Hardware Resale. NHR employs more than 200 people in the United States and Europe. Sheldon discussed methods for identifying fake gear, the refurbished hardware industry's efforts to educate users about counterfeits and Cisco's role in eradicating this growing problem.
The news has been full of reports of counterfeit gear lately - particularly Cisco gear - and the potential harm it could cause. As you might imagine, the refurbished hardware industry has developed a lot of expertise at detecting fake gear.
Network Hardware Resale is not an authorized Cisco reseller nor is it affiliated with Cisco in any way. Mike has been an outspoken personality in the whole area of fighting counterfeits and has plenty of insight to share, but his opinions are his own. Prior to NHR, Mike worked for nine years in investment banking for UBS and Barclays Capital in New York.
Are there counterfeit cables being sold?
Not to my knowledge - but keep in mind there are a lot of generic cables being manufactured and sold, and they are functionally identical to the Cisco-branded cables, perfectly legitimate for Cisco customers to use, and also very inexpensive. Given that the main difference will be the presence or absence of the Cisco logo on the head of the cable, a counterfeit cable would certainly be possible to make, but I've not heard of them - hard to make a living selling $4 cables.
It seems clear that counterfeits are mostly identified by failure of components. How else are counterfeits identified? I am interested to hear how Cisco and the hardware industry in general are addressing the counterfeiting problem. Other industries, like software, are using embedded digital signals/signatures; the pharmaceutical world thinks that it is addressing counterfeiting with mass serialization and RFIDs.
Cisco has evolved from simple serial number stickers many years ago to hologram stickers on many pieces of equipment today, as well as embedded Common Language Equipment Identification (CLEI) codes, IDPROMs, some of which are proprietary to Cisco, that must match data on the exterior of the unit itself. Some Cisco products reject components that do not have the correct internal codes.
Where are most of the counterfeits coming from?
Almost all comes from Asia, China being the overwhelming culprit.
There has been a lot of articles written about the FBI's Cisco Raider sting. Cisco Gold partners were caught selling counterfeits into government accounts. How can this happen? I thought only gray market companies sold counterfeit.
Just so everyone is one the same page, "Cisco Raider" is the FBI's name of an ongoing multi-agency counter-counterfeiting initiative. Yes, the main culprit in the FBI's quietly circulated PowerPoint was counterfeit sellers (many from eBay) selling to GSA-approved vendors who would then resell on to the government, and many of these GSA vendors were Cisco authorized partners. The channel community sells boxes, the secondary market sells individually used pieces of hardware. I think if you deal with a trustworthy channel partner, who buys from Cisco directly, you are totally covered - but good secondary market vendors are no more the source of counterfeits than Ingram or Comstor. It's the low-end bottom feeders that are the issue.
Why aren't you cooperating with Cisco? Cisco and other manufacturers say the only way to be sure of authenticity is to buy from their authorized channels. How do you respond to that? How can you guarantee the quality of second-hand equipment? I've heard that if I buy used I get no warranty or technical support?
We would love to cooperate with Cisco, but with few exceptions Cisco has refused our numerous overtures. In NHR's case specifically, Cisco has twice offered to help with one specific instance of suspected counterfeit hardware, but the terms under which it offered its help were unacceptable, and its offer did not include any education or explanation at all - just a simple yes it's authentic, or no it's counterfeit. We would also have been required to supply Cisco with complete information about every purchase from the same vendor over time.
I do want to reiterate an offer that NHR and the United Network Equipment Dealer Association (UNEDA) has made directly to Cisco and in print a number of times: We would welcome the opportunity to cooperate with Cisco and eradicate the counterfeit problem forever. We will sign an NDA, we will travel to you, we will meet at your convenience. Working together we can make the resale of counterfeit goods so difficult, and so unprofitable, that the purveyors of counterfeit hardware will be put out of business.
Also, the counterfeit resellers depend on cheap, public, widely viewed marketing to sell their products (like eBay). Get them off of eBay, Alibaba, and the like, and much of the problem goes away.
Regarding buying authorized, where do the authorized resellers get their equipment from? If it's shipped directly from Cisco that's certainly one way to be sure - but Cisco Raider showed that it's the source, not the reseller, that is the issue. Also, if a piece of solid-state hardware, designed to work for decades is $10,000 from an authorized seller and $2,500 from the secondary - and is guaranteed to be authentic - I think most companies would agree there are real reasons to look outside the channel.
The reasons for looking to the secondary market for equipment also go beyond price, and include end-of-life gear, emergency procurement needs (the secondary market can supply things in days - from Cisco it often takes weeks or months), better standard warranties, and significantly cheaper maintenance costs.
Warranty and support is another area of confusion regarding secondary market hardware. Cisco offers a variety of standard warranties on its hardware, but most are basic repair-and-replace at best, and do not include tech support.
Secondary market vendors have had to exceed this in order to compete, and most offer 90-day to 1-year or longer. (NHR offers 1-year advance replacement on everything we sell), as well as add-on tech support offering 24-hour support and next-day to 4-hour response at a cost that is a fraction of Cisco SmartNet.
What tools are available which allow the detection of counterfeits and are any of them IP enabled?
The Cisco CCO tool allows someone to check the validity of a serial number, but good counterfeits use real serials. Besides this, the tools are mostly "preponderance of evidence" based, as we are not capable of definitively proving something to be authentic - only deeming it suspicious - and if we are suspicious (using over a dozen criteria) we reject it.
How are counterfeits identified? Failures? Other methods? (Using RFIDs or other methods?)
WICs, VICs, VWICs, NM Cards, and GBICs/SFPs are the overwhelming majority of counterfeit Cisco that we see. For a VWIC card for example, there are some very specific things to look for: 1) A valid serial number (check the Cisco CCO), 2) A correct serial number sticker (legit stickers are bar-code readable, and the bar code generally goes to the edge of the sticker itself, rather than being printed in the middle) that has a laminated surface (counterfeits often print the bar code directly on the plastic sticker, and it will rub off to the touch); 3) internal and printed serial numbers that match; 4) A Stewart RJ-45 jack that protrudes about 1/16th of an inch (counterfeits are often mounted flush or have a non-Stewart jack); 5) In a 'show diag', no visible errors, i.e., "FRU ... unknown," "RMA ... unknown," symbols used in the hex code, etc. We use about seven more softer criteria as well, and compare any suspect product to known good samples. There are many more "tricks of the trade" that vary by specific product.
Mike - I understand that most of the "new" WIC, PA and NM cards on eBay are actually fake/counterfeit. I complained to my eBay rep but they tell me that only Cisco has the right to complain because it is the "intellectual rights" owner. It looks like Cisco is totally ignoring the hundreds of fake items on eBay. Why do you think Cisco is ignoring this problem?
Type WIC-1ENET into eBay and look at the products sold, $15 each and "new in bulk pack", and at least one seller boasts of its "premier Cisco partner" status right on the posting. FYI, Cisco does not sell WICs in bulk packaging and my average cost used is over $60 each. So you are right, eBay's VERO program depends on the IP owner - Cisco - to police the site. I honestly do not know why these are allowed to exist - perhaps being able to sow fear and uncertainty about the secondary market is more valuable than eliminating the problem.
How big do you think the counterfeit market is for Cisco and is this a bigger problem for Cisco or the secondary market?
There were a lot of counterfeit products on the US market six or seven years ago. I think today, as long as you define the "market" as the upper end secondary market dealers - members of UNEDA for example, which I'll talk more about in a second - the presence of counterfeits is almost nil. eBay, Alibaba (which Cisco invested in) and other online sites are the main purveyors. So I think it's really a problem for Cisco. Channel partners typically end up with counterfeit equipment when they look to online auctions like eBay as a source for "new" equipment in an effort to reduce their cost - a practice NHR would not recommend to anyone. Channel partners are resellers of new equipment, not used gear, so it stands to reason that many are not equipped to screen equipment for authenticity. Their typical approach would be to receive a product, not open the packaging, and forward it along to a customer. If it is truly new direct from Cisco, that approach would be adequate, but when you are purchasing from other sources, rigorous testing is necessary.
For perspective, in Q1 of this year we received more than 35,000 items into our facilities worldwide and 0.45 per cent were quarantined as suspicious (including counterfeit, re-manufactured or badly repaired, or otherwise not a match to established standards).
I've heard at the end of July CCO is being retired, so therefore you won't be able to check serials there anymore. What then?
I have heard that only a Cisco partner will be able to check serials, and for them only a contract issued through them to the customer will show up - but again, as I am not Cisco, this is only what I've heard. We will have to wait and see. From the perspective of identifying counterfeits, serials are not terribly useful as good counterfeits have valid serials - the other criteria are more important.
What do you think of embedding other "covert" security information in the bar code to make the bar code more secure?
That's outside my area of expertise, but something like that - and if the identification of such were shared with the vendor community - it could help a lot, assuming they could be made difficult to counterfeit themselves.
Doesn't Cisco end up with the bad end of the deal if these items are covered by SmartNet and fail?
Well, no more than we, or other vendors, do who warranty them ourselves. That's why I would say that our interests and Cisco's are truly aligned when it comes to counterfeits.
Do you think there would be any resistance to needing a separate reader to "read" the (non-duplicable) covert security info?
I would buy one certainly...
Are any purchases from NHR considered authorized for SmartNet maintenance by Cisco? Or are they all subject to an inspection by Cisco prior to being eligible?
Cisco's policy states that any secondary market item may require an inspection and/or a relicensing prior to being eligible for SmartNet. This is similar to Cisco's intellectual property language that states customers "may" be violating Cisco's IP if they do not relicense a product. We have seen Cisco partners wave one or both requirements, and we have seen Cisco partners require both. NHR (among others) also offers its own 24-hour, NBD maintenance on all Cisco hardware at a fraction of the cost of SmartNet, so there are options.
Mike, do you foresee a day when Cisco would ever work with the non-authorized reseller market to combat counterfeit product?
Oh, I think so, and certainly hope so. If we worked together, closely monitored the auction sites and other online channels, worked with law enforcement (which we do already) I think we could end the problem in under a year. If the counterfeiters have no buyers, they will move on to other things. Through a combination of education and monitoring, we could eliminate their channel.
What kinds of gear is most commonly counterfeit these days?
GBICS and SFPs, WIC, VWIC, VIC, and NM cards are the most common. But it's important to realize that the efforts of the secondary market and the FBI and others seem to be working - even within these products we see many fewer counterfeit cards than in 2002.
Cisco is able to maintain price margins by forcing end users to return product using the allegation that the product is counterfeit. Does that happen to NHR on certain big sales to resellers? If so, how does NHR respond?
We hear that Cisco often tells customers products "may" be counterfeit, and almost always talks about the risks associated with products not being licensed and thereby "may" violate Cisco's IP. We have had a few deals returned, but much more often we are able to talk the customer through what is really going on - a highly competitive fight for a deal - and we end up fine. If the issue is potential counterfeits we demand proof as we are certain they are not. More often it is not "Cisco" saying this but Cisco salespeople, and their statements are often nothing more than extreme flights of fantasy - once called on it they have to back off.
How much of a security problem do you think counterfeit gear might be? I can think of several scenarios where the counterfeit gear allows someone from the outside to spy on targets who have installed what they thought was legitimate equipment.
I don't see any evidence of that, nor have we ever seen anything - hardware or software - installed on a Cisco router for these purposes - it is certainly possible, but we haven't seen it.
Your opinion? Cisco wants to keep a manageable amount of counterfeit gear on the market as a tool to keep concerned end users asking the question "Why aren't you cooperating with Cisco?" and to keep them asking the question "How can I be sure of authenticity unless the product is purchased through a Cisco authorized channel?"
I wouldn't bet against it, but really I hope it's just negligence. The presence of counterfeits does offer Cisco a great FUD tool in its competition with the secondary market, but the Cisco Raider report shows the fallacy of that argument. The right answer is to buy from those who can tell the difference, and have been doing so for years, rather than to buy from anyone with a Cisco partnership (like that premier partner on eBay selling WIC-1ENET for $15).
But it is not in Cisco's interest to clear up the counterfeiting, right? It allows Cisco to cast doubt on the market and maintain absurd margins (especially with the government). Otherwise Cisco would respond to bona fide industry groups/players (as other OEMs do), instead of demanding all info then refusing cooperation. Have you asked the government to look into why Cisco won't cooperate?
I've heard a lot of good thoughts along those lines. Cisco is very arguably a monopoly in the networking market, and that brings with it - or should - a certain responsibility. When its margins on memory and GBICS are something like 90 per cent+ and overall margins around 65 per cent+ (I think it's actually 67 per cent?), there are good financial reasons to use whatever tools available to push people to the channel. What is true is that Cisco makes great products that last a long time. It is inevitable, though frustrating for Cisco, that these will be (and SHOULD BE) reused. They are inexpensive, functional, supported, and even a bit green.
It would appear that Cisco is intentionally blurring the lines between counterfeit and legitimate secondary market product to intimidate customers into buying only from the channel. I would think that Cisco would help protect its customers by clearly identifying what measures it could take to protect themselves besides buying new.
I agree with you 100 per cent, as did the FBI in its presentation. The best remedy would seem to be education and cooperation with a very loyal support group for their products - the secondary market. Keep in mind as well that if its recertification fees made any sense - they are absurd - and it allowed us to easily put our products on SmartNet, Cisco stands to gain tens of millions of dollars in revenue.
Are other vendors' gear are also being faked or is it really just Cisco?
Many others - computer NIC cards for example. I know IBM, HP, and others are involved in combating the problem.
You mentioned that Alibaba is a site where counterfeit items are sold. Cisco made a huge investment in Alibaba (I think close to 20 million), so why would it invest in a site that promotes counterfeit Cisco equipment? Furthermore, why would it continue to manufacture in China?
That is a great question. They are no harder to find there then they are on eBay, and many are the very same sellers. Regarding where to manufacture, I can't imagine it's more complicated than cost, but that's a good question for Cisco.
What is your opinion on used equipment dealers gathering together, sharing information to help combat the counterfeit problem?
I'm glad you asked. We have! Many secondary market Cisco vendors have banded together to form a trade association called UNEDA (http://www.uneda.com/) whose sole purpose is to ensure the quality of and promote the value of secondary market products. Our top initiative is counterfeit identification and education for members, and we expel any member who is found to knowingly have resold counterfeit goods.
Without Cisco's help, how accurate are our attempts to discern what is real from what isn't? I feel as if we are chasing our tail on some of the lower end items. Holograms, perpendicular logos, approved vendors, even serial numbers used to be good indicators of real product, but now they are easily being counterfeited. We figure something out today and tomorrow the counterfeiters are able to replicate it. Feels like the blind trying to help the blind? What's your take?
We cannot guarantee authenticity, but we can cast doubt on it - and we err on the side of caution. If there are suspicious markings or signs, we pass on the item. We could be less intense about it, but we really have only our name and reputation to use in our competition with Cisco for business. The last thing I would want is some inadvertent mistake to give Cisco more ammunition. We just need to be very careful.
Do counterfeit parts or components cause damage to a server or network?
I've never heard of anything like that. I've heard of badly repaired items causing problems, but no counterfeit items actually causing physical damage.
We have been offered supposedly new "factory sealed" equipment at discount levels approaching 50 per cent off of normal Cisco list price. How is this possible on the gray market?
Cisco gives discounts to many customers, and on certain deals, better than 50 per cent off. We also purchase a huge quantity of overstock new items from Cisco partners and Cisco customers who lost a deal, cancelled a project, etc. - in fact about 30 per cent of my current inventory is "new in box" surplus gear.
Has this UNEDA tried to contact Cisco at any level about working with Cisco to eradicate counterfeiting?
We have, in three certified letters and in numerous overtures in the press. (Including here - we really do want to meet!)
Cisco, and many manufacturers for that matter, say that by educating each other we are adding to the counterfeit problem. This seems to me a way for them to paint counterfeiters and used equipment dealers with the same brush.
There is a common belief in the secondary market that Cisco likes to use the grey = black message, and it is really unfortunate. Nothing could be further from the truth. The secondary market is overwhelmingly good people making a living reselling a good product.
Another concern Cisco has with grey marketers is with the license transfers. I've read many grey market case laws and the first-sales doctrine that support the grey market trade, however it's still a little unclear. Would you be able to expand?
Long answer required - but yes, our strongly held belief is that embedded/included software transfers with the sale of a product. If this were not the case our whole industry would not exist, but neither would eBay and most other resale businesses. Cisco carefully words its position with phrases like "may violate" but in my opinion it's pretty transparently toothless. There are many cases that have supported the transfer of embedded IP and suffice it to say in such a limited venue that the transfer of such IP does not seem to be an issue.
What are some of the best tactics to prevent the buying/selling of counterfeit?
Don't buy off eBay, buy from a reputable and well-known reseller, and demand a good warranty with indefinite return.
Mike, what do you feel is the biggest hurdle in partnering/aligning the used resellers with Cisco in an effort to combat counterfeit gear?
I really don't know. We are ready and willing to meet. I believe there is a very strongly held belief that grey = black, but we should really be viewed no differently than Juniper. We compete but can also cooperate. I hope it happens.
With your maintenance options, how would we get software updates and how would we know if Cisco requires an inspection? If a SmartNet contract is created, does that guarantee that Cisco will honor the contract?
With a valid license you can generally get bug patches and such, but for major revision updates you need SmartNet, or you need to re-buy a license. Keep in mind that the vast majority of products do not require updates, like fixed configuration switches, and as equipment ages most customers standardize on a given revision and rarely deviate.
So, for a company that's considering the used market for the first time (times are tough), what key questions should we be asking the vendor?
How long have you been in business? What warranty do you offer? What technical support capabilities do you have (number of certified technicians, hours support is available)? Also ask if the vendor stocks and tests everything it sells prior to selling it. With the right vendor you can save a lot of money, and I think you'll find the customer service, support, warranty, unbiased advice, and responsiveness exceed your expectations. Give it a try!