Is it possible to build 5k$ (VAT not included) raid matrix PC with 36TB storage? What configuration would you suggest?
You can buy white box cases with 16 or 24 bays from from outfits like Supermicro. An example of this type of case is a Supermicro 846E1-R1200B, which has a 1200w PSU, holds 24 drives and goes for around £1,000 here in the UK. It may be a bit cheaper in the U.S.
RAID Controller and disks
The case I mentioned has an expander backplane, which means you will only need 4-6 SATA channels. An 8-channel controller like an Adaptec ASR-4800S or 3Ware 9550XLP (or a PCI-e equivalent) can be purchased off ebay for a few hundred dollars, and you should be able to get 20 or so 2GB disks for less than $100 each. Get a reputable controller that does RAID-50 or RAID-60 as 20 disks is a bit much for a single RAID-5. Find out the part number for the cache battery on the controller and buy a shiny new one. Again, Ebay is your friend, and you can probably get one for less (often much less) than $100.
If you have enough left in your budget it might be worth getting a spare RAID controller of the same type for DR purposes.
Motherboard and CPU
Look at the web sites for Asus, Tyan, Supermicro or Gigabyte to find a suitable server motherboard with the right type of slots. Older 2-socket boards are quite cheap on Ebay and might be worth considering, even if you only plan to populate one socket. Boards of this type can accept a lot of memory - often 32GB, 64GB or more. Having a lot of memory for disk buffers is usually a performance win for a file server so the large memory capacity is potentially a useful feature even if you don't initially intend to populate it.
Opterons, in particular, have fallen out of fashion now so an older Opteron motherboard can be bought off Ebay for a few hundred dollars and CPU chips are cheaper than Xeons. An example of such a board is the Tyan S29xx series (which takes DDR2 memory and IIRC Socket AM2 chips), or the older S28xx series that takes Socket 940 chips and DDR1 memory. A quick search on ebay turns up 340 hits for 'Tyan Opteron Motherboard' available to the U.S.
DDR2 memory is a bit of a sweet spot as there was a glut on the market when this was current. Modules of this type (DDR2 registered ECC) are quite readily available off ebay with a street price of $200-300 (as of late 2010) for an 8GB kit. Note that there are a few different flavours (Xeon boards of the era often use FB DIMMS) so make sure you get the right type for your motherboard.
Older Socket 940 boards such as the S28xx family take DDR1 memory and 4GB DDR1 modules are quite rare and expensive. Using 4GB kits (2x2GB modules - typically installed in pairs) you can normally put up to 16GB on a motherboard of this type. After that, DDR2 will get cheaper. There are also a couple of generations of Xeon boards that might be relevant, but Xeon kit tends to hold more of its value and the parts will be dearer.
With a bit of judicial shopping you should be able to build a box of this spec for something close to your $5,000 budget. This approach gets you a server grade motherboard (a few years old, but presumably with decent caps), a server grade redundant PSU, a pukka RAID controller and a rack mountable case with room and airflow for your disks.
A few caveats
Well cheapo 2TB consumer disks are about £$€80 each, you can get a dual core PC for £$€300, so you just need a couple of JBOD eSATA controllers and however many disk enclosures you want to carve it up in (I'd suggest 4 x 5 disk boxes for price). Stick an OS on it and away you go.
Crappy way of doing it perhaps but if money is the key issue you limit your choices.
Maybe Backblaze pod (google it) is worth considering, there are some guys that will build case for you, add OpenSolaris with two SSDs for performance and off you go :)
There is more in this article: ZFS - Building, Testing, and Benchmarking