Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I am working with 2 x Supermicro SC837E26-RJBOD1 chassis connected to a single LSI 9285-8e card in a Supermicro 1U host. There are 28 drives in each chassis for a total of 56 drives in 28 RAID1 mirrors.

The problem I am running in to is that there are duplicate slots for the 2 chassis (the slots list twice and only go from 0 to 27). All the drives also show the same enclosure ID (ID 36). However, MegaCLI -encinfo lists the 2 enclosures correctly (ID 36 and ID 65).

My question is, why would this happen? Is there an option I am missing to use 2 enclosures effectively?

This is blocking me rebuilding a drive that failed in slot 11 since I can only specify enclosure and slot as parameters to replace a drive. When I do this, it picks the wrong slot 11 (device ID 46 instead of device ID 19).

Adapter #1 is the LSI 9285-8e, adapter #0 (which I removed due to space limitations) is the onboard LSI.

Adapter information:

Adapter #1

==============================================================================
                    Versions
                ================
Product Name    : LSI MegaRAID SAS 9285-8e
Serial No       : SV12704804
FW Package Build: 23.1.1-0004

                    Mfg. Data
                ================
Mfg. Date       : 06/30/11
Rework Date     : 00/00/00
Revision No     : 00A
Battery FRU     : N/A

                Image Versions in Flash:
                ================
BIOS Version       : 5.25.00_4.11.05.00_0x05040000
WebBIOS Version    : 6.1-20-e_20-Rel
Preboot CLI Version: 05.01-04:#%00001
FW Version         : 3.140.15-1320
NVDATA Version     : 2.1106.03-0051
Boot Block Version : 2.04.00.00-0001
BOOT Version       : 06.253.57.219

                Pending Images in Flash
                ================
None

                PCI Info
                ================
Vendor Id       : 1000
Device Id       : 005b
SubVendorId     : 1000
SubDeviceId     : 9285

Host Interface  : PCIE

ChipRevision    : B0

Number of Frontend Port: 0
Device Interface  : PCIE

Number of Backend Port: 8
Port  :  Address
0        5003048000ee8e7f
1        5003048000ee8a7f
2        0000000000000000
3        0000000000000000
4        0000000000000000
5        0000000000000000
6        0000000000000000
7        0000000000000000

                HW Configuration
                ================
SAS Address      : 500605b0038f9210
BBU              : Present
Alarm            : Present
NVRAM            : Present
Serial Debugger  : Present
Memory           : Present
Flash            : Present
Memory Size      : 1024MB
TPM              : Absent
On board Expander: Absent
Upgrade Key      : Absent
Temperature sensor for ROC    : Present
Temperature sensor for controller    : Absent

ROC temperature : 70  degree Celcius

                Settings
                ================
Current Time                     : 18:24:36 3/13, 2012
Predictive Fail Poll Interval    : 300sec
Interrupt Throttle Active Count  : 16
Interrupt Throttle Completion    : 50us
Rebuild Rate                     : 30%
PR Rate                          : 30%
BGI Rate                         : 30%
Check Consistency Rate           : 30%
Reconstruction Rate              : 30%
Cache Flush Interval             : 4s
Max Drives to Spinup at One Time : 2
Delay Among Spinup Groups        : 12s
Physical Drive Coercion Mode     : Disabled
Cluster Mode                     : Disabled
Alarm                            : Enabled
Auto Rebuild                     : Enabled
Battery Warning                  : Enabled
Ecc Bucket Size                  : 15
Ecc Bucket Leak Rate             : 1440 Minutes
Restore HotSpare on Insertion    : Disabled
Expose Enclosure Devices         : Enabled
Maintain PD Fail History         : Enabled
Host Request Reordering          : Enabled
Auto Detect BackPlane Enabled    : SGPIO/i2c SEP
Load Balance Mode                : Auto
Use FDE Only                     : No
Security Key Assigned            : No
Security Key Failed              : No
Security Key Not Backedup        : No
Default LD PowerSave Policy      : Controller Defined
Maximum number of direct attached drives to spin up in 1 min : 10
Any Offline VD Cache Preserved   : No
Allow Boot with Preserved Cache  : No
Disable Online Controller Reset  : No
PFK in NVRAM                     : No
Use disk activity for locate     : No

                Capabilities
                ================
RAID Level Supported             : RAID0, RAID1, RAID5, RAID6, RAID00, RAID10, RAID50, RAID60, PRL 11, PRL 11 with spanning, SRL 3 supported, PRL11-RLQ0 DDF layout with no span, PRL11-RLQ0 DDF layout with span
Supported Drives                 : SAS, SATA

Allowed Mixing:

Mix in Enclosure Allowed
Mix of SAS/SATA of HDD type in VD Allowed

                Status
                ================
ECC Bucket Count                 : 0

                Limitations
                ================
Max Arms Per VD          : 32
Max Spans Per VD         : 8
Max Arrays               : 128
Max Number of VDs        : 64
Max Parallel Commands    : 1008
Max SGE Count            : 60
Max Data Transfer Size   : 8192 sectors
Max Strips PerIO         : 42
Max LD per array         : 16
Min Strip Size           : 8 KB
Max Strip Size           : 1.0 MB
Max Configurable CacheCade Size: 0 GB
Current Size of CacheCade      : 0 GB
Current Size of FW Cache       : 887 MB

                Device Present
                ================
Virtual Drives    : 28
  Degraded        : 0
  Offline         : 0
Physical Devices  : 59
  Disks           : 56
  Critical Disks  : 0
  Failed Disks    : 0

                Supported Adapter Operations
                ================
Rebuild Rate                    : Yes
CC Rate                         : Yes
BGI Rate                        : Yes
Reconstruct Rate                : Yes
Patrol Read Rate                : Yes
Alarm Control                   : Yes
Cluster Support                 : No
BBU                             : No
Spanning                        : Yes
Dedicated Hot Spare             : Yes
Revertible Hot Spares           : Yes
Foreign Config Import           : Yes
Self Diagnostic                 : Yes
Allow Mixed Redundancy on Array : No
Global Hot Spares               : Yes
Deny SCSI Passthrough           : No
Deny SMP Passthrough            : No
Deny STP Passthrough            : No
Support Security                : No
Snapshot Enabled                : No
Support the OCE without adding drives : Yes
Support PFK                     : Yes
Support PI                      : No
Support Boot Time PFK Change    : Yes
Disable Online PFK Change       : No
PFK TrailTime Remaining         : 0 days 0 hours
Support Shield State            : Yes
Block SSD Write Disk Cache Change: Yes

                Supported VD Operations
                ================
Read Policy          : Yes
Write Policy         : Yes
IO Policy            : Yes
Access Policy        : Yes
Disk Cache Policy    : Yes
Reconstruction       : Yes
Deny Locate          : No
Deny CC              : No
Allow Ctrl Encryption: No
Enable LDBBM         : No
Support Breakmirror  : No
Power Savings        : Yes

                Supported PD Operations
                ================
Force Online                            : Yes
Force Offline                           : Yes
Force Rebuild                           : Yes
Deny Force Failed                       : No
Deny Force Good/Bad                     : No
Deny Missing Replace                    : No
Deny Clear                              : No
Deny Locate                             : No
Support Temperature                     : Yes
Disable Copyback                        : No
Enable JBOD                             : No
Enable Copyback on SMART                : No
Enable Copyback to SSD on SMART Error   : Yes
Enable SSD Patrol Read                  : No
PR Correct Unconfigured Areas           : Yes
Enable Spin Down of UnConfigured Drives : Yes
Disable Spin Down of hot spares         : No
Spin Down time                          : 30
T10 Power State                         : Yes
                Error Counters
                ================
Memory Correctable Errors   : 0
Memory Uncorrectable Errors : 0

                Cluster Information
                ================
Cluster Permitted     : No
Cluster Active        : No

                Default Settings
                ================
Phy Polarity                     : 0
Phy PolaritySplit                : 0
Background Rate                  : 30
Strip Size                       : 64kB
Flush Time                       : 4 seconds
Write Policy                     : WB
Read Policy                      : Adaptive
Cache When BBU Bad               : Disabled
Cached IO                        : No
SMART Mode                       : Mode 6
Alarm Disable                    : Yes
Coercion Mode                    : None
ZCR Config                       : Unknown
Dirty LED Shows Drive Activity   : No
BIOS Continue on Error           : No
Spin Down Mode                   : None
Allowed Device Type              : SAS/SATA Mix
Allow Mix in Enclosure           : Yes
Allow HDD SAS/SATA Mix in VD     : Yes
Allow SSD SAS/SATA Mix in VD     : No
Allow HDD/SSD Mix in VD          : No
Allow SATA in Cluster            : No
Max Chained Enclosures           : 16
Disable Ctrl-R                   : Yes
Enable Web BIOS                  : Yes
Direct PD Mapping                : No
BIOS Enumerate VDs               : Yes
Restore Hot Spare on Insertion   : No
Expose Enclosure Devices         : Yes
Maintain PD Fail History         : Yes
Disable Puncturing               : No
Zero Based Enclosure Enumeration : No
PreBoot CLI Enabled              : Yes
LED Show Drive Activity          : Yes
Cluster Disable                  : Yes
SAS Disable                      : No
Auto Detect BackPlane Enable     : SGPIO/i2c SEP
Use FDE Only                     : No
Enable Led Header                : No
Delay during POST                : 0
EnableCrashDump                  : No
Disable Online Controller Reset  : No
EnableLDBBM                      : No
Un-Certified Hard Disk Drives    : Allow
Treat Single span R1E as R10     : No
Max LD per array                 : 16
Power Saving option              : Don't Auto spin down Configured Drives
Max power savings option is  not allowed for LDs. Only T10 power conditions are to be used.
Default spin down time in minutes: 30
Enable JBOD                      : No
TTY Log In Flash                 : No
Auto Enhanced Import             : No
BreakMirror RAID Support         : No
Disable Join Mirror              : No
Enable Shield State              : Yes
Time taken to detect CME         : 60s

Exit Code: 0x00

Enclosure information:

# /opt/MegaRAID/MegaCli/MegaCli64 -encinfo -a1

    Number of enclosures on adapter 1 -- 3

    Enclosure 0:
    Device ID                     : 36
    Number of Slots               : 28
    Number of Power Supplies      : 2
    Number of Fans                : 3
    Number of Temperature Sensors : 1
    Number of Alarms              : 1
    Number of SIM Modules         : 0
    Number of Physical Drives     : 28
    Status                        : Normal
    Position                      : 1
    Connector Name                : Port B
    Enclosure type                : SES
    VendorId is LSI CORP and Product Id is SAS2X36
VendorID and Product ID didnt match
    FRU Part Number               : N/A
    Enclosure Serial Number       : N/A
    ESM Serial Number             : N/A
    Enclosure Zoning Mode         : N/A
    Partner Device Id             : 65

    Inquiry data                  :
        Vendor Identification     : LSI CORP
        Product Identification    : SAS2X36
        Product Revision Level    : 0718
        Vendor Specific           : x36-55.7.24.1

Number of Voltage Sensors         :2

Voltage Sensor                    :0
Voltage Sensor Status             :OK
Voltage Value                     :5020 milli volts

Voltage Sensor                    :1
Voltage Sensor Status             :OK
Voltage Value                     :11820 milli volts

Number of Power Supplies     : 2

Power Supply                 : 0
Power Supply Status          : OK

Power Supply                 : 1
Power Supply Status          : OK

Number of Fans               : 3

Fan                          : 0
Fan Speed              :Low Speed
Fan Status                   : OK

Fan                          : 1
Fan Speed              :Low Speed
Fan Status                   : OK

Fan                          : 2
Fan Speed              :Low Speed
Fan Status                   : OK

Number of Temperature Sensors : 1

Temp Sensor                  : 0
Temperature                  : 48
Temperature Sensor Status    : OK

Number of Chassis             : 1

Chassis                      : 0
Chassis Status               : OK

    Enclosure 1:
    Device ID                     : 65
    Number of Slots               : 28
    Number of Power Supplies      : 2
    Number of Fans                : 3
    Number of Temperature Sensors : 1
    Number of Alarms              : 1
    Number of SIM Modules         : 0
    Number of Physical Drives     : 28
    Status                        : Normal
    Position                      : 1
    Connector Name                : Port A
    Enclosure type                : SES
    VendorId is LSI CORP and Product Id is SAS2X36
VendorID and Product ID didnt match
    FRU Part Number               : N/A
    Enclosure Serial Number       : N/A
    ESM Serial Number             : N/A
    Enclosure Zoning Mode         : N/A
    Partner Device Id             : 36

    Inquiry data                  :
        Vendor Identification     : LSI CORP
        Product Identification    : SAS2X36
        Product Revision Level    : 0718
        Vendor Specific           : x36-55.7.24.1

Number of Voltage Sensors         :2

Voltage Sensor                    :0
Voltage Sensor Status             :OK
Voltage Value                     :5020 milli volts

Voltage Sensor                    :1
Voltage Sensor Status             :OK
Voltage Value                     :11760 milli volts

Number of Power Supplies     : 2

Power Supply                 : 0
Power Supply Status          : OK

Power Supply                 : 1
Power Supply Status          : OK

Number of Fans               : 3

Fan                          : 0
Fan Speed              :Low Speed
Fan Status                   : OK

Fan                          : 1
Fan Speed              :Low Speed
Fan Status                   : OK

Fan                          : 2
Fan Speed              :Low Speed
Fan Status                   : OK

Number of Temperature Sensors : 1

Temp Sensor                  : 0
Temperature                  : 47
Temperature Sensor Status    : OK

Number of Chassis             : 1

Chassis                      : 0
Chassis Status               : OK

    Enclosure 2:
    Device ID                     : 252
    Number of Slots               : 8
    Number of Power Supplies      : 0
    Number of Fans                : 0
    Number of Temperature Sensors : 0
    Number of Alarms              : 0
    Number of SIM Modules         : 1
    Number of Physical Drives     : 0
    Status                        : Normal
    Position                      : 1
    Connector Name                : Unavailable
    Enclosure type                : SGPIO
Failed in first Inquiry commnad
    FRU Part Number               : N/A
    Enclosure Serial Number       : N/A
    ESM Serial Number             : N/A
    Enclosure Zoning Mode         : N/A
    Partner Device Id             : Unavailable

    Inquiry data                  :
        Vendor Identification     : LSI
        Product Identification    : SGPIO
        Product Revision Level    : N/A
        Vendor Specific           :


Exit Code: 0x00

Now, notice that each slot 11 device shows an enclosure ID of 36, I think this is where the discrepancy happens. One should be 36. But the other should be on enclosure 65.

Drives in slot 11:

Enclosure Device ID: 36
Slot Number: 11
Drive's postion: DiskGroup: 5, Span: 0, Arm: 1
Enclosure position: 0
Device Id: 48
WWN:
Sequence Number: 11
Media Error Count: 0
Other Error Count: 0
Predictive Failure Count: 0
Last Predictive Failure Event Seq Number: 0
PD Type: SATA
Raw Size: 2.728 TB [0x15d50a3b0 Sectors]
Non Coerced Size: 2.728 TB [0x15d40a3b0 Sectors]
Coerced Size: 2.728 TB [0x15d400000 Sectors]
Firmware state: Online, Spun Up
Is Commissioned Spare : YES
Device Firmware Level: A5C0
Shield Counter: 0
Successful diagnostics completion on :  N/A
SAS Address(0): 0x5003048000ee8a53
Connected Port Number: 1(path0)
Inquiry Data:       MJ1311YNG6YYXAHitachi HDS5C3030ALA630                 MEAOA5C0
FDE Enable: Disable
Secured: Unsecured
Locked: Unlocked
Needs EKM Attention: No
Foreign State: None
Device Speed: 6.0Gb/s
Link Speed: 6.0Gb/s
Media Type: Hard Disk Device
Drive Temperature :30C (86.00 F)
PI Eligibility:  No
Drive is formatted for PI information:  No
PI: No PI
Drive's write cache : Disabled
Drive's NCQ setting : Enabled
Port-0 :
Port status: Active
Port's Linkspeed: 6.0Gb/s
Drive has flagged a S.M.A.R.T alert : No



Enclosure Device ID: 36
Slot Number: 11
Drive's postion: DiskGroup: 19, Span: 0, Arm: 1
Enclosure position: 0
Device Id: 19
WWN:
Sequence Number: 4
Media Error Count: 0
Other Error Count: 0
Predictive Failure Count: 0
Last Predictive Failure Event Seq Number: 0
PD Type: SATA
Raw Size: 2.728 TB [0x15d50a3b0 Sectors]
Non Coerced Size: 2.728 TB [0x15d40a3b0 Sectors]
Coerced Size: 2.728 TB [0x15d400000 Sectors]
Firmware state: Online, Spun Up
Is Commissioned Spare : NO
Device Firmware Level: A580
Shield Counter: 0
Successful diagnostics completion on :  N/A
SAS Address(0): 0x5003048000ee8e53
Connected Port Number: 0(path0)
Inquiry Data:       MJ1313YNG1VA5CHitachi HDS5C3030ALA630                 MEAOA580
FDE Enable: Disable
Secured: Unsecured
Locked: Unlocked
Needs EKM Attention: No
Foreign State: None
Device Speed: 6.0Gb/s
Link Speed: 6.0Gb/s
Media Type: Hard Disk Device
Drive Temperature :30C (86.00 F)
PI Eligibility:  No
Drive is formatted for PI information:  No
PI: No PI
Drive's write cache : Disabled
Drive's NCQ setting : Enabled
Port-0 :
Port status: Active
Port's Linkspeed: 6.0Gb/s
Drive has flagged a S.M.A.R.T alert : No

Update 06/28/12:

I finally have some new information about (what we think) the root cause of this problem so I thought I would share.

After getting in contact with a very knowledgeable Supermicro tech, they provided us with a tool called Xflash (doesn't appear to be readily available on their FTP). When we gathered some information using this utility, my colleague found something very strange:

root@mogile2 test]# ./xflash.dat -i get avail

Initializing Interface.
Expander: SAS2X36 (SAS2x36)


1) SAS2X36 (SAS2x36) (50030480:00EE917F)  (0.0.0.0) 
2) SAS2X36 (SAS2x36) (50030480:00E9D67F)  (0.0.0.0) 
3) SAS2X36 (SAS2x36) (50030480:0112D97F)  (0.0.0.0)

This lists the connected enclosures. You see the 3 connected (we have since added a 3rd and a 4th which is not yet showing up) with their respective SAS address / WWN (50030480:00EE917F). Now we can use this address to get information on the individual enclosures:

[root@mogile2 test]# ./xflash.dat -i 5003048000EE917F get exp 

Initializing Interface.
Expander: SAS2X36 (SAS2x36)


Reading the expander information..........

            Expander: SAS2X36 (SAS2x36) B3
         SAS Address: 50030480:00EE917F
Enclosure Logical Id: 50030480:0000007F
          IP Address: 0.0.0.0
Component Identifier: 0x0223
  Component Revision: 0x05

[root@mogile2 test]# ./xflash.dat -i 5003048000E9D67F get exp                 

Initializing Interface.
Expander: SAS2X36 (SAS2x36)


Reading the expander information..........

            Expander: SAS2X36 (SAS2x36) B3
         SAS Address: 50030480:00E9D67F
Enclosure Logical Id: 50030480:0000007F
          IP Address: 0.0.0.0
Component Identifier: 0x0223
  Component Revision: 0x05

[root@mogile2 test]# ./xflash.dat -i 500304800112D97F get exp                 

Initializing Interface.
Expander: SAS2X36 (SAS2x36)


Reading the expander information..........

            Expander: SAS2X36 (SAS2x36) B3
         SAS Address: 50030480:0112D97F
Enclosure Logical Id: 50030480:0112D97F
          IP Address: 0.0.0.0
Component Identifier: 0x0223
  Component Revision: 0x05

Did you catch it? The first 2 enclosures logical ID is partially masked out where the 3rd one (which has a correct unique enclosure ID) is not. We pointed this out to Supermicro and were able to confirm that this address is supposed to be set during manufacturing and there was a problem with a certain batch of these enclosures where the logical ID was not set.

We believe that the RAID controller is determining the ID based on the logical ID and since our first 2 enclosures have the same logical ID, they get the same enclosure ID. We also confirmed that 0000007F is the default which comes from LSI as an ID.

The next pointer that helps confirm this could be a manufacturing problem with a run of JBODs is the fact that all 6 of the enclosures that have this problem begin with 00E. I believe that between 00E8 and 00EE Supermicro forgot to program the logical IDs correctly and neglected to recall or fix the problem post production.

Fortunately for us, there is a tool to manage the WWN and logical ID of the devices from Supermicro: ftp://ftp.supermicro.com/utility/ExpanderXtools_Lite/. Our next step is to schedule a shutdown of these JBODs (after data migration) and reprogram the logical ID and see if it solves the problem.

Update 06/28/12 #2:

I just discovered this FAQ at Supermicro while Google searching for "lsi 0000007f": http://www.supermicro.com/support/faqs/faq.cfm?faq=11805. I still don't understand why, in the last several times we contacted Supermicro, they would have never directed us to this article :\

share|improve this question
    
Can you paste in some output from the megacli output? –  Tom O'Connor Mar 12 '12 at 21:29
    
Can you start the rebuild from the controller during POST as a work-around? –  MikeAWood Mar 13 '12 at 0:16
    
Tom, I added some MegaCLI output to outline the issue. Mike, nope! The machine is in heavy production and this is obviously the last thing I want to try... –  Andy Shinn Mar 14 '12 at 1:46
    
I had the same problem with another enclosure, 847E26. The 846E16 as well as the 836E16 didn't present this problem, the 2 backplanes don't share the same SAS enclosure ID. I don't know for sure if this is a misconfiguration on the E26 or a more general misconception, but I asked for an RMA after that Supermicro support ignored my requests for help for weeks. –  wazoox Mar 19 '12 at 16:01
    
Adding this information as it may be relevant. I believe the chip used in the Supermicro enclosures is lsi.com/products/storagecomponents/Pages/LSISAS2x28.aspx. I don't actually see a single 28 port PHY. So I am guessing this could be the chip used in both the E26 and E16 variants. If this is true, I don't see why the distinction between E26 or E16 should cause the problem. But this is very helpful information none the less! I am still continuing to solve this issue as it just recently caused a major headache for us. –  Andy Shinn Jun 26 '12 at 0:56
add comment

2 Answers

up vote 2 down vote accepted

We managed to finally fix this issue. The ultimate cause and fix? It seems a manufacturing process error caused some JBODs shipped from Supermicro to come with a default burned in logical ID (the 0000007F). This address is actually supposed to match the SAS address by default.

To fix this issue, we had to run a tool called ExpanderXtools Lite (ftp://supermicro.com/utility/ExpanderXtools_Lite/). You run the SMC binary and will get a X window popup (you'll either need X installed, or if you are not running X on your servers like us, a local X server on your laptop being forwarded through SSH). In the SMC program you select the COM menu and click on inband.

Now, you can go to the WWN menu and select WWN. A new popup will show your JBOD primary and secondary (if you have the E26 model) controllers. You need to update both controllers at the same time before closing the window. After updated and the window is closed, power off the array for a bit and power back on. Use the SMC binary again to verify the logical address shows correctly.

The biggest pain point of this was having to power down the arrays. It might be possible to do this online and do a rescan with your RAID card. But better to play it safe. Port IDs for disks will change. For us, our LSI card was able to pick up the arrays after the change. Your mileage may vary.

share|improve this answer
    
(First time seeing this Q, sorry I couldn't help earlier) SuperMicro is somewhat infamous for screwing up Logical IDs, MACs, and any sort of unique ID. Fortunately LSI makes tools to renumber the screwed up devices, so it's usually a quick fix. –  Chris S Nov 15 '12 at 0:13
    
I have exactly the same problem! Even asked question here, but in my case xflash throws double buffering error in glibc and cannot change anything. What OS/distro and version did you use? –  neoX Apr 2 at 21:25
    
I was using CentOS 6 on the host machines containing the LSI card. –  Andy Shinn Apr 3 at 15:56
add comment

I don't know, why MegaRAID have in slot listing for both enclosures the same ID, but you definitely can identify this two enclosures by Connected Port Number.

Enclosure listing:

ID36 is on the adapter port B
ID63 is on the adapter port A

Slot 11 listing:

Device ID48 is on the port 1
Device ID19 is on the port 0

I think, that you can identify port 0 as port A and port 1 as port B

share|improve this answer
    
Is there a way for MegaCLI to rebuild an array using something other than slot and enclosure? The biggest issue here is that an array with a degraded drive in the second chassis cannot be rebuilt because it always selects slot 11 on chassis ID 36, –  Andy Shinn Mar 16 '12 at 5:23
1  
@AndyShinn Did you try only remove this drive and give it back after a minute? It is a operation, which should initiate rebuild process... –  Jan Marek Mar 16 '12 at 5:36
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.