Cahiers techniques

The best way to transfer a huge amount of data?

iapx — Sat, 19 Sep 2009 01:28:00 GMT

I read an article about Google's research on what's the bandwidth on data transfers using different means. And they ended-up putting a bunch of big hard-drive into a box to transfer it overnight from one location to the other :-)

The same way we will shutdown a File Sharing website (a huge one, 4000 Mb/s bandwidth!), and I wanted to get a bunch of files, 500GB of them. I began to download them from work, using our 8Mb (1MB/s) internet connection, but today I switched to another strategy...

I select the files from web interface, mark them, and ask one of our Download server to store it onto one 2 TB hard-drive, and I plan to have all the files I am looking for on monday or tuesday, and then just remove he hard-drive and copy it at home :-)

Using our web access, it may take me 40 work days, checking things regularly and loossing my time. With this option, I will just let a daemon to the job for me, and assume a real incredible bandwidth, 36 hours pour 500GB is 30 Mb/s continous bandwidth, 3X better than anything I could expect from my Internet Provider, and really 6X-9X faster in real life :-)

The better is the server will work for me, and my computer will stay asleep!

PS: Eric, please don't ask about the files, you probably know what is it! loooollllllll

7 millions visitors!

iapx — Sat, 27 Jun 2009 22:39:00 GMT

Whatever the size of your website, even if you have a single page blog, statistics are important to show you if you are on the right way or not.

We were under 6 millions visitors per month when I came, 3 months ago, and now largely over 7 millions visitors, I hope to hit the 10 million this year :-)

The key was to have a global vision, technical as well as business-oriented.
The technology is the key to implement the business vision, not a goal on itself.

PS: We deliver the quivalent of 50 000 full-length movie each and every day!

Proprietary solution...

iapx — Tue, 16 Jun 2009 15:24:09 GMT

Each time I discover another feature on our Isilon iQ 6000 it is deceptive.

For example when you replace an hard drive that is faulty, the reconstruction is all but automagic, you have to enter "isi devices" command by hand on the console, with a risk of human-error (ie: not formating the new hard-drive but another). Terrific!

Moreover, the "famous" OneFS file system that span across a cluster of 5 boxes and 60 hard-drives, that deliver the bandwidth of one good SSD drive on a bsic PC (sic), seems not to be able to use bigger hard-drive when you have a mix, so 80's!

I changed one of the consumer-grade 500GB 7200rpm Hitachi Deskstar that came within a node (they only use consumer grade hard-drive, not server-grade, the server-grade you will have it on the billing price of an Isilon cluster not on it's component!), that has failed and replaced it by a 1000GB (1TB) 7200rpm SATA hard-drive, and guess what?
The capacity remains the same in the node, only recognizing it as a 500GB hard-drive with 5 logical partitions.

Does OneFS be limited to one-size hard-drive on a box or on a cluster?
Even desktop NAS haven't this kind of limitation anymore, spreading datas (and copies) on many drives of different sizes.

Does ISILON limits usage of their Cluster to force customer to buy new ones when one is full?
(instead of buying inexpensive high-capacity hard-drives and raising 2X to 4X the capacity of their existing system)

Anyway, OneFS & ISILON is a proprietary system, with programmed-obsolescence...

ISILON & Spare-Hard drive

iapx — Thu, 16 Apr 2009 19:23:00 GMT

We bought spare hard-drives for our ISILON Cluster (5x ISILON IQ 6000i with Infiniband switch).

When I opened the box, I was surprised: 500GB Hitachi Deskstar consumer-grade hard-drives, that are sold for less than 80$ usually!!!

Worse, they are in their Hitachi's packaging, ISILON doesn't even unpack them before shipping to test them, so you could end up putting a 500GB spare hard-drive in a server and discover that this drive isn't operational!

Amateur? Who said amateur???

ISILON OneFS & Adding/deleting files problemSISILON,OneFS

iapx — Wed, 15 Apr 2009 20:21:00 GMT

When you use a classical RAID-5 system (or whatever RAID you use), adding files to the system or deleting them (clean-up) isn't a problem in itself, and doesn't change the performance-level of the system after the file being put on the disk.

On ISILON OneFS, when you add files, and even when you delete them, it launch a restripe process that run in the background and eat as much bandwidth as it could (hard-drive & memory) to reconstruct a new "safe" structure.

The result is that on a Production Server, if you add many files or even do a good clean-up, the performances drop significantly, impairing user/customer experience, and dragging down our website.

This is one of the major drawback of the OneFS ISILON file system. Maybe it's good to know!

ISILON IQ 6000i & actual models

iapx — Sun, 12 Apr 2009 15:17:00 GMT

We own a cluster of 5 IQ 6000i, that costed us more than 100 000$, and I would like to describe what is the hardware of each box (20 000$+).

- Server-grade Motherboard
- Pentium 4 Xeon CPU (actual model use Core2 Quad 8200 consumer-grade CPU)
- 4GB DDR RAM
- 12 SATA Hitachis Deskstart consumer-grade hard-drive.

All that for 20 000$+.

On the performance-level, the hard-drive may deliver 8 000Mb/s streaming, or 3600Mb/s with our 2MB random read (our payload is 2MB read at-once, not-so-randomly), and the box deliver 600Mb/s (1/6th) crashing at peak.

I won't say it's too expensive, I prefer to say that it doesn't deliver the expected performance level, at least 900Mb/s on 1x1Gb ethernet link, or 1800Mb/s (50% of practical hard-drive performances) on 2x1Gb Ethernet links.

It just doesn't deliver, at 20 000$+ per box

PS: Naturally our cluster use "infiniband" that is far from infinite bandwidth and is also a limiting factor, even with 5 boxes.

ISILON cuts 10% of staff

iapx — Sun, 12 Apr 2009 02:33:19 GMT

Read the article here

ISILON IQ 6000i & high payload

iapx — Sun, 12 Apr 2009 01:56:09 GMT

I upgraded the whole web server, including revising firewall balance between servers, and now we have a big problem with our 100 000$+ cluster of ISILON IQ 6000i:

As shown in the graph, it just crashed yesterday, and worse the only way to retrieve control was to shut it down and on, as the SSH Shell wasn't responding, only the file service, degraded.
I checked if there's a disk problem, and all 12 disks are flagged up and running before the restart.

Today again, as you could see, the performance was degraded during 3 hours, before this box retrieve it's performance level. A 20 000$+ box that offers a 600Mb file service, something I would expect from my desktop PC with it's 3 disks, or from our 2 000$ 1U server box with 3 hard-drives. Without crashing.

The ISILON IQ 6000i is really unreliable: doesn-t support high payload, and worse crash when the payload is too high! The only solution seems to buy a 15 000$ support contract (that won't make it faster) or add other boxes for 30 000$+++

I will replace it with PC Server boxes loaded with big hard-drives, that are reliable and faster.

ISILON IQ 6000i & OneFS

iapx — Sat, 04 Apr 2009 17:39:29 GMT

At my new job (CTO of a big filesharing website), we use a cluster of 5 ISILON IQ 6000i as our main storage, 27TB clustered and theorically redundant...

ISILON pretend their solution is clustered, it's true, they pretend that when a drive fail, the reconstruction is faster than other solution, it's only partially true, they pretend it's better than other NAS, and that's totally wrong!

I will talk about that later. Maybe there's things you should now about OneFS, that you wont find anywhere on internet ...

smarter : part 2

iapx — Sun, 01 Mar 2009 02:48:00 GMT

The SQL DB Administrator is smart, maybe the smarter guy (in his mind), because he could explain why he's leak-speak may not be found by anybody.

You know a password like "gros beta" became " Gr0sb3t4.". Something that any hacker from 10 to 25 did since he use a keyboard! That's not a security, because any password use the same pattern, it just limit the number of possibilities :-)

First letter in upper case, last letter is lacking or is a sign, leak transcoding is predictable, so you have two choices, use dictionnary attack, around 2^40 possibilities, or full password scan, around 2^48 possibilities.
The first case is fast, under 1-day using actual Core 2 Duo laptop, the second option may takes 1 to 4 weeks using Core2 Duo and nVidia's CUDA integrated GPU (says GeForce 8600GT), and I know for sure the password doesn't changed since months :-)

Yes, he's smart, he's using a pattern that is internally known, and we know for sur 80% of attacks comme from inside (it's statistical), so this translate in limiting the seqarch-range and the time to have a successfull attack using modern CUDA technologies! Thanks MB to be so smart :-)