Questions tagged [big-data]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
0 votes
0 answers
57 views

YARN resource manager "HTTP request sent, awaiting response..."

we have Hadoop cluster with active/stand by resource manager services the active resource manager is on master1 machine and the stand by resource manager is on master2 machine in our cluster YARN ...
0 votes
1 answer
4k views

how to determine yarn.scheduler.maximum-allocation-vcores value in ambari cluster

we have ambari cluster ( version 2.6 ) with 3 workers machine , and each worker machine have 16 CPU CORE ( see pic down ) , while each machine have 32G memory according to: yarn.nodemanager.resource....
1 vote
0 answers
173 views

HDP cluster + journal nodes get out of Sync

we have HDP cluster version 2.6.5 when we look on name-node logs we can see the following warning 2023-02-20 15:56:37,731 INFO namenode.FileJournalManager (FileJournalManager.java:finalizeLogSegment(...
7 votes
2 answers
17k views

Are we able to edit the schema of bigquery table after creation?

I made a mistake of specifying a field as integer instead of float. I found that I am not able to make correction a field once the table is created. I have to delete and re-create the table again to ...
0 votes
1 answer
2k views

Spark-Cassandra-Connector Issue Exception in thread "main" java.lang.NoClassDefFoundError: com/datastax/spark/connector/rdd/reader/RowReaderFactory

What is going wrong with the Spark Cassandra Connector could you please help to solve this? Scala File: import com.datastax.spark.connector._ import org.apache.spark.sql.SparkSession import org.apache....
-2 votes
1 answer
93 views

How do megasites like Youtube perform backups?

How do megasites like Youtube perform backups? According to https://www.quora.com/Where-does-YouTube-store-so-many-videos, 2014 they stored 76 PB every year, a number that most certainly has increased ...
0 votes
1 answer
721 views

HDFS balancing , how to balanced hdfs data?

we have Hadoop version - 2.6.4 On the datanode machine we can see that hdfs data isn’t balanced On some disks we have different used size as sdb 11G and sdd 17G /dev/sdd 20G 3.0G 17G 15% /grid/...
10 votes
7 answers
10k views

How does one check huge files identity if hashing is CPU bound?

For small files hashing is just ok, but with huge ones you can easily find md5sum is CPU bound. Is there any hashing algorithm able to scale out on multiple cores? Any workarounds? Ideas? Anything? :)
4 votes
1 answer
341 views

Reclaiming free space in filegroup with single chronological partition

-moved here from SO (no coments there) Question: what is a proper way of reclaiming space in big (hundreds of GBs) filegroup with single partition of table that is chronologically ordered and has no ...
0 votes
0 answers
96 views

Avoid kafka disk to became 100% used by Cron job

We want to suggest the following based on our issues on kafka disks We have many HDP clusters ( based on ambari , and all machines are redhat version 7.2 ) Each cluster include 3 kafka machines , ...
0 votes
1 answer
806 views

why ambari agent insist to create another repository file

we are installing the new hadoop version - 2.6.3.0 on ambari - 2.6.0 from ambari agent log we see the follwing: Writing File['/etc/yum.repos.d/ambari-hdp-51.repo'] because contents don't match why ...
1 vote
0 answers
63 views

spark.dynamicAllocation + setting the spark parameters according to ambari cluster

we want to find the values for the following spark parameters according to inputs as memory on datanode machine , CPU CORE on data node machine , numbers of datanode machine etc ,, spark....
0 votes
1 answer
489 views

How to reconfigure Ambari services values with blueprint.json file

we have many Ambari LAB clusters - Apache Ambari Version 2.5.0.3 , while ambari agent installed on Linux redhat machines my target is to find a way to update the values of services , on all the ...
1 vote
0 answers
142 views

Presto Maximum concurrent sessions

Presto can't handle many concurrent sessions. what's the maximum number of concurrent sessions per presto and how to set parameter for this ? and how to handle it's maximum JVM ?
3 votes
2 answers
16k views

What is the best way for store images of website? [closed]

We are using cassandra database for store website information, but we are not sure how to save images. We can store them in cassandra, but we can also allocate a server for storing images. ...
1 vote
1 answer
2k views

Cloudera SCM Agent can't heartbeat but port is contactable

I'm trying to add nodes to a Cloudera cluster. When the agent starts I get a python stacktrace saying it can't heartbeat to master-host:7182, however I can connect to that port just fine. The ...
0 votes
2 answers
158 views

At what point do you consider moving from the cloud to colocation?

I'm currently operating at a cost of about $25k - $40k per month on AWS. I have about 30TB of data indexed in Elasticsearch, running a 4 node production cluster, and another 4 node staging cluster. ...
0 votes
0 answers
969 views

Quickest way to get large number of small files from remote FTP server

In Fintech, the following scenario seems fairly common: You've paid for access to a huge collection of data, but it is made available to you as thousands of little files, each with a footprint in ...
2 votes
0 answers
3k views

Cassandra nodetool repair failure - broken pipe

we're trying to check our Cassandra cluster data integrity with: nodetool repair but after several minutes (~2-10min), we got strange connection resets / broken pipe stack trace on a first node: ...
1 vote
0 answers
335 views

Range requests time out after 10GB in apache or java servlet

There seems to be an issue with serving large 10+GB files with byte-range requests on our RHEL5 64-bit server. The issue that I am noticing is that range requests are timing out for ranges that are ...
1 vote
0 answers
2k views

HDFS performances on apache spark

I have several issues related to HDFS, that may have different roots. I'm posting as much information as I can, with the hope that I can get your opinion on at least some of them. Basically the cases ...
3 votes
1 answer
453 views

Apache Spark infrastructure - combining compute and storage nodes

I have an infrastructure question around Apache Spark, which I'm looking at rolling out in a greenfield project with (at most) approximately 4 TB of data used for modelling at any given time. ...
1 vote
1 answer
59 views

Is there a way to send (by mail) external disks to be installed in azure?

We have many TBs of data on external disks (WD passports), and wish to process it using azure's VMs. Uploading will take forever (and the bandwidth will probably cost too much). Is there a way ...
0 votes
1 answer
375 views

MySQL Cluster ndb_restore fails without error

I have been working to migrate our current single instance database to a new clustered database running MySQL cluster. It is a large database (several billion records) and, while it seems to be ...
-2 votes
1 answer
103 views

0.5 TB SQL Server Database, is it possible to store in a standalone server? [closed]

We are facing the task of storing of 0.5 TB of data in a SQL Server 2008 Server. Is it possible to do it in a standalone server? Later we also want to query for generating statistics of the data (a ...
0 votes
0 answers
128 views

Hadoop - On the Wire Performance Monitoring?

I have been tasked with implementing an 'on the wire' monitoring solution for a large Hadoop installation. The source of data will be a combination of taps and SPANs throughout the environment. My ...
0 votes
1 answer
656 views

mhddfs not support single file to split multiple hard drives.. if file size exceed limit of single storage device

I'm using mhddfs to combine multiple drives that are mounted over network using NFS. e.g. There are three machines Server Name Dir Space Server 1 /home 10 GB Space Server 2 /home 10 GB ...
2 votes
2 answers
341 views

Just how bad is Network Attached Storage for certain cloud applications?

I have heard it recommended to stay away from AWS hosting for certain "big data" applications (e.g. Hadoop, Cassandra, Solr) because EC2 instances typically use network attached storage (though there ...