Questions tagged [big-data]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
0
votes
0answers
35 views

How to copy files from HDFS recursive to the local file system

How we can copy recursive jar files from HDFS ( jar files are under sub folders ) to local folder? Example export hdfs_folder=/app/lib export local_folder=/home/work_app while under /app/lib we ...
0
votes
1answer
31 views

HDFS balancing , how to balanced hdfs data?

we have Hadoop version - 2.6.4 On the datanode machine we can see that hdfs data isn’t balanced On some disks we have different used size as sdb 11G and sdd 17G /dev/sdd 20G 3.0G 17G 15% /grid/...
0
votes
0answers
42 views

Avoid kafka disk to became 100% used by Cron job

We want to suggest the following based on our issues on kafka disks We have many HDP clusters ( based on ambari , and all machines are redhat version 7.2 ) Each cluster include 3 kafka machines , ...
0
votes
1answer
584 views

how to determine yarn.scheduler.maximum-allocation-vcores value in ambari cluster

we have ambari cluster ( version 2.6 ) with 3 workers machine , and each worker machine have 16 CPU CORE ( see pic down ) , while each machine have 32G memory according to: yarn.nodemanager.resource....
1
vote
0answers
40 views

spark.dynamicAllocation + setting the spark parameters according to ambari cluster

we want to find the values for the following spark parameters according to inputs as memory on datanode machine , CPU CORE on data node machine , numbers of datanode machine etc ,, spark....
0
votes
1answer
454 views

why ambari agent insist to create another repository file

we are installing the new hadoop version - 2.6.3.0 on ambari - 2.6.0 from ambari agent log we see the follwing: Writing File['/etc/yum.repos.d/ambari-hdp-51.repo'] because contents don't match why ...
3
votes
1answer
1k views

Are we able to edit the schema of bigquery table after creation?

I made a mistake of specifying a field as integer instead of float. I found that I am not able to make correction a field once the table is created. I have to delete and re-create the table again to ...
0
votes
1answer
279 views

How to reconfigure Ambari services values with blueprint.json file

we have many Ambari LAB clusters - Apache Ambari Version 2.5.0.3 , while ambari agent installed on Linux redhat machines my target is to find a way to update the values of services , on all the ...
1
vote
0answers
116 views

Presto Maximum concurrent sessions

Presto can't handle many concurrent sessions. what's the maximum number of concurrent sessions per presto and how to set parameter for this ? and how to handle it's maximum JVM ?
0
votes
2answers
135 views

At what point do you consider moving from the cloud to colocation?

I'm currently operating at a cost of about $25k - $40k per month on AWS. I have about 30TB of data indexed in Elasticsearch, running a 4 node production cluster, and another 4 node staging cluster. ...
0
votes
0answers
455 views

Quickest way to get large number of small files from remote FTP server

In Fintech, the following scenario seems fairly common: You've paid for access to a huge collection of data, but it is made available to you as thousands of little files, each with a footprint in ...
7
votes
7answers
3k views

How does one check huge files identity if hashing is CPU bound?

For small files hashing is just ok, but with huge ones you can easily find md5sum is CPU bound. Is there any hashing algorithm able to scale out on multiple cores? Any workarounds? Ideas? Anything? :)
1
vote
1answer
962 views

Cloudera SCM Agent can't heartbeat but port is contactable

I'm trying to add nodes to a Cloudera cluster. When the agent starts I get a python stacktrace saying it can't heartbeat to master-host:7182, however I can connect to that port just fine. The ...
2
votes
0answers
2k views

Cassandra nodetool repair failure - broken pipe

we're trying to check our Cassandra cluster data integrity with: nodetool repair but after several minutes (~2-10min), we got strange connection resets / broken pipe stack trace on a first node: ...
1
vote
0answers
192 views

Range requests time out after 10GB in apache or java servlet

There seems to be an issue with serving large 10+GB files with byte-range requests on our RHEL5 64-bit server. The issue that I am noticing is that range requests are timing out for ranges that are ...
1
vote
0answers
2k views

HDFS performances on apache spark

I have several issues related to HDFS, that may have different roots. I'm posting as much information as I can, with the hope that I can get your opinion on at least some of them. Basically the cases ...
3
votes
1answer
397 views

Apache Spark infrastructure - combining compute and storage nodes

I have an infrastructure question around Apache Spark, which I'm looking at rolling out in a greenfield project with (at most) approximately 4 TB of data used for modelling at any given time. ...
1
vote
1answer
49 views

Is there a way to send (by mail) external disks to be installed in azure?

We have many TBs of data on external disks (WD passports), and wish to process it using azure's VMs. Uploading will take forever (and the bandwidth will probably cost too much). Is there a way ...
4
votes
1answer
156 views

Reclaiming free space in filegroup with single chronological partition

-moved here from SO (no coments there) Question: what is a proper way of reclaiming space in big (hundreds of GBs) filegroup with single partition of table that is chronologically ordered and has no ...
0
votes
1answer
290 views

MySQL Cluster ndb_restore fails without error

I have been working to migrate our current single instance database to a new clustered database running MySQL cluster. It is a large database (several billion records) and, while it seems to be ...
-2
votes
1answer
92 views

0.5 TB SQL Server Database, is it possible to store in a standalone server? [closed]

We are facing the task of storing of 0.5 TB of data in a SQL Server 2008 Server. Is it possible to do it in a standalone server? Later we also want to query for generating statistics of the data (a ...
0
votes
0answers
124 views

Hadoop - On the Wire Performance Monitoring?

I have been tasked with implementing an 'on the wire' monitoring solution for a large Hadoop installation. The source of data will be a combination of taps and SPANs throughout the environment. My ...
0
votes
1answer
489 views

mhddfs not support single file to split multiple hard drives.. if file size exceed limit of single storage device

I'm using mhddfs to combine multiple drives that are mounted over network using NFS. e.g. There are three machines Server Name Dir Space Server 1 /home 10 GB Space Server 2 /home 10 GB ...
2
votes
2answers
327 views

Just how bad is Network Attached Storage for certain cloud applications?

I have heard it recommended to stay away from AWS hosting for certain "big data" applications (e.g. Hadoop, Cassandra, Solr) because EC2 instances typically use network attached storage (though there ...
3
votes
2answers
12k views

What is the best way for store images of website? [closed]

We are using cassandra database for store website information, but we are not sure how to save images. We can store them in cassandra, but we can also allocate a server for storing images. ...