Take the 2-minute tour ×
Server Fault is a question and answer site for system and network administrators. It's 100% free, no registration required.

We are running ColdFusion 8 on a windows server 2003 VPS with an API that exposes student records to a partner API through a connector. Our API returns around 50k student records serialized in XML format pretty seamlessly. My question originates when something very frightening happened today when we tested our connector to our partners API. Our entire website and web host went down. We assumed that our host was just having some issues and after 4 hours with no resolution and no response from their customer service we finally got a response from them claiming that they had an "unauthorized user" in their network. After our server was back up we were unable to connect to our website as if the web service or coldfusion itself had froze. This is really where my concern comes from as I fear we may have overloaded the web service.

As I mentioned before we tried sending over 50k HTTP POST requests over to our partner's API, however everything stopped after around 1.6k

Is this bad practice or is there some sort of rate limiting I can relax somewhere in server configuration? We managed to find a workaround, but it bypasses our connector which is critical to our design.

This would have been a one time deal as the purpose of so many requests was to populate our partner's website with current data, after that hourly syncs will keep requests down to around 100 per hour.

UPDATE

Our partner API is owned and operated by Pardot. We are converting students to prospects by passing student data to their API which unfortunately only seems to accept one student at a time. For that reason we have to do all 50k requests individually.

Our server has 4GB of RAM, an Intel Core 2 Duo @ 2.8GHz running Windows Server 2003 SP2.

I monitored the server during a 100 student sync, a 400 student sync, and a 1.4k student sync with the following results:

100 students - 2.25GB of Memory, 30-40% CPU utilization, 0.2-0.3% network bandwidth

400 students - 2.30GB of Memory, 30-50% CPU utilization, 0.2-1.0% network bandwidth

1.4k students - 2.30GB of Memory, 30-70% CPU utilization, 0.2-1.0% network bandwidth

I know this is a far cry from 50k students, but I don't want to risk taking down our CMS system again assuming that was the cause.

To give you a look at our code:

<cfif (#getStudents.statusCode# eq "200 OK")>   
    <cftry>
        <cfloop index="StudentXML" array="#XmlSearch(responseSTUD,'/students/student')#">
            <cfset StudentXML = XmlParse(StudentXML)>
            <cfhttp
                url="#PARDOT_CMS_UPSERT#"
                method="post" 
                timeout="10000" >
                <cfhttpparam type="url" name="user_key" value="#PARDOT_CMS_USERKEY#">
                <cfhttpparam type="url" name="api_key" value="#api_key#">
                <cfhttpparam type="url" name="email" value="#StudentXML.student.email.XmlText#">
                <cfhttpparam type="url" name="first_name" value="#StudentXML.student.first.XmlText#">
                <cfhttpparam type="url" name="last_name" value="#StudentXML.student.last.XmlText#">
                <cfhttpparam type="url" name="in_cms" value="#StudentXML.student.studentid.XmlText#">
                <cfhttpparam type="url" name="company" value="#StudentXML.student.agencyname.XmlText#">
                <cfhttpparam type="url" name="country" value="#StudentXML.student.countryname.XmlText#">
                <cfhttpparam type="url" name="address_one" value="#StudentXML.student.address.XmlText#">
                <cfhttpparam type="url" name="address_two" value="#StudentXML.student.address2.XmlText#">
                <cfhttpparam type="url" name="city" value="#StudentXML.student.city.XmlText#">
                <cfhttpparam type="url" name="state" value="#StudentXML.student.state_province.XmlText#">
                <cfhttpparam type="url" name="zip" value="#StudentXML.student.postalcode.XmlText#">
                <cfhttpparam type="url" name="phone" value="#StudentXML.student.phone.XmlText#">
                <cfhttpparam type="url" name="fax" value="#StudentXML.student.fax.XmlText#">
                <cfhttpparam type="url" name="output" value="simple">
            </cfhttp>
        </cfloop>
    <cfcatch type="any">
        <cfdump var="#cfcatch.Message#">
    </cfcatch>
    </cftry>
</cfif>

UPDATE 2

I checked the CF logs and found a couple of these:

"Error","jrpp-8","06/06/13","16:10:18","CMS-API","Java heap space The specific sequence   of files included or processed is: D:\Clients\www.xxx.com\www\dev.cms\api\v1\api.cfm, line: 675 "
java.lang.OutOfMemoryError: Java heap space
at java.util.Arrays.copyOf(Arrays.java:2882)
at java.io.CharArrayWriter.write(CharArrayWriter.java:105)
at coldfusion.runtime.CharBuffer.replace(CharBuffer.java:37)
at coldfusion.runtime.CharBuffer.replace(CharBuffer.java:50)
at coldfusion.runtime.NeoBodyContent.write(NeoBodyContent.java:254)
at         cfapi2ecfm292155732._factor30(D:\Clients\www.xxx.com\www\dev.cms\api\v1\api.cfm:675)
at   cfapi2ecfm292155732._factor31(D:\Clients\www.xxx.com\www\dev.cms\api\v1\api.cfm:662)
at cfapi2ecfm292155732._factor36(D:\Clients\www.xxx.com\www\dev.cms\api\v1\api.cfm:659)
at cfapi2ecfm292155732._factor42(D:\Clients\www.xxx.com\www\dev.cms\api\v1\api.cfm:657)
at cfapi2ecfm292155732._factor37(D:\Clients\www.xxx.com\www\dev.cms\api\v1\api.cfm)
at cfapi2ecfm292155732._factor44(D:\Clients\www.xxx.com\www\dev.cms\api\v1\api.cfm:456)
at cfapi2ecfm292155732._factor38(D:\Clients\www.xxx.com\www\dev.cms\api\v1\api.cfm)
at cfapi2ecfm292155732._factor46(D:\Clients\www.xxx.com\www\dev.cms\api\v1\api.cfm:455)
at cfapi2ecfm292155732._factor39(D:\Clients\www.xxx.com\www\dev.cms\api\v1\api.cfm)
at cfapi2ecfm292155732._factor47(D:\Clients\www.xxx.com\www\dev.cms\api\v1\api.cfm:453)
at cfapi2ecfm292155732.runPage(D:\Clients\www.xxx.com\www\dev.cms\api\v1\api.cfm:1)
at coldfusion.runtime.CfJspPage.invoke(CfJspPage.java:192)
at coldfusion.tagext.lang.IncludeTag.doStartTag(IncludeTag.java:366)
at coldfusion.filter.CfincludeFilter.invoke(CfincludeFilter.java:65)
at coldfusion.filter.ApplicationFilter.invoke(ApplicationFilter.java:279)
at coldfusion.filter.RequestMonitorFilter.invoke(RequestMonitorFilter.java:48)
at coldfusion.filter.MonitoringFilter.invoke(MonitoringFilter.java:40)
at coldfusion.filter.PathFilter.invoke(PathFilter.java:86)
at coldfusion.filter.ExceptionFilter.invoke(ExceptionFilter.java:70)
at coldfusion.filter.ClientScopePersistenceFilter.invoke(ClientScopePersistenceFilter.java:28)
at coldfusion.filter.BrowserFilter.invoke(BrowserFilter.java:38)
at coldfusion.filter.NoCacheFilter.invoke(NoCacheFilter.java:46)
at coldfusion.filter.GlobalsFilter.invoke(GlobalsFilter.java:38)
at coldfusion.filter.DatasourceFilter.invoke(DatasourceFilter.java:22)
at coldfusion.CfmServlet.service(CfmServlet.java:175)
at coldfusion.bootstrap.BootstrapServlet.service(BootstrapServlet.java:89)
at jrun.servlet.FilterChain.doFilter(FilterChain.java:86)

Looks like I might have crashed the JVM in CF, is there a better way to do this? We are thinking of just exporting all records initially as a CSV file and importing it into Pardot seeing as we will never have to do a request this large again.

share|improve this question
1  
I'd initially suspect a full java heap. What JVM settings are you using? Are you running all 50k requests from a single request being executed in coldfusion? Can you turn on JMX remoting to take a look at what the JVM is doing? –  Shane Madden Jun 7 '13 at 6:01
    
I would have to get back to you on that, I don't have access to our CFAdmin, but we are making each request individually –  Jason Bristol Jun 7 '13 at 12:14
    
You were right, the heap was full, looking into alternative solutions –  Jason Bristol Jun 7 '13 at 12:45

2 Answers 2

Have you monitored CPU, disk and bandwidth usage on your server? The first thing I'd do is make sure none of these resources are being overloaded. Perfmon is an obvious place to start and check out your CF logs.

Were you making 50K post requests (i.e. separate requests) one after the other or is this a single request for the 50K records in an XML file? If the former then why would you make 50K requests? Could it be done by requesting all 50K records at once as that would be just 1 request and 1 file transfer? If the 50K requests are sequential (i.e. the next request isn't made until the current one is complete) then that seems like a gentle way of making a large number of requests.

share|improve this answer
    
Hmm I didn't think of that, I will check out our server stats during a request tomorrow and post them here. We are making one request after another sequentially. Our API spits out one huge XML file containing all students that gets parsed and then each student is POSTed to our partners API. Currently I don't think their API will accept an XML request, but I could look into it as it would make this a lot easier –  Jason Bristol Jun 7 '13 at 0:49
    
I updated my question with the information you suggested –  Jason Bristol Jun 7 '13 at 12:05
up vote 0 down vote accepted

I found solution that pretty much bypasses this issue and falls more in line of a "best practice"

I decided to go the route of exporting data to a CSV file since this issue stemmed from trying to initially populate a database. CSV import works flawlessly and afterwords syncs only issue around 0-1000 requests, with some further optimization and scheduling I have requests down to 0-50 per 30 minutes which is ideal.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.