lrnz.name

User Tools

Site Tools

Trace: • dokuwiki • playground • python • big-data-analytics • ssh • development • software
big-data-analytics

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Next revision		Previous revision
big-data-analytics [2017-11-21 10:13] – created mrksbig-data-analytics [2026-04-22 10:54] (current) – mrks
Line 1: Line 1:
 ====== Big Data Analytics ====== ====== Big Data Analytics ======
 +
 +
 +
 +===== Fundamentals =====
 +
 +
 +  * [[https://en.wikipedia.org/wiki/Character_encoding|Character encodings]]
 +    * [[https://en.wikipedia.org/wiki/US-ASCII|ASCII]]
 +    * [[https://en.wikipedia.org/wiki/ISO/IEC_8859-1|ISO 8859-1]]
 +    * [[https://en.wikipedia.org/wiki/Unicode|Unicode]]
 +    * [[https://en.wikipedia.org/wiki/UTF-8|UTF-8]]
 +    * [[https://en.wikipedia.org/wiki/UTF-16|UTF-16]]
 +    * [[https://en.wikipedia.org/wiki/UTF-32|UTF-32]]
 +    * [[https://en.wikipedia.org/wiki/EBCDIC|EBCDIC]]
 +  * [[https://en.wikipedia.org/wiki/File_format|File formats]]
 +    * [[https://en.wikipedia.org/wiki/Comma-separated_values|CSV]]
 +    * [[https://en.wikipedia.org/wiki/XML|XML]] and [[https://en.wikipedia.org/wiki/XML_Schema_(W3C)|XSD]]
 +    * [[https://en.wikipedia.org/wiki/JSON|JSON]]
 +  * [[https://en.wikipedia.org/wiki/Database|Databases]]
 +    * [[https://en.wikipedia.org/wiki/SQL|SQL]]
 +    * [[https://en.wikipedia.org/wiki/Java_Database_Connectivity|JDBC]]
 +  * [[https://en.wikipedia.org/wiki/Computer_network|Network]]
 +    * [[https://en.wikipedia.org/wiki/IPv4|IPv4]] and [[https://en.wikipedia.org/wiki/Subnetwork|network mask]]
 +    * [[https://en.wikipedia.org/wiki/IPv6|IPv6]]
 +    * [[https://en.wikipedia.org/wiki/IP_routing|Routing]] and [[https://en.wikipedia.org/wiki/Router_(computing)|Router]]
 +    * [[https://en.wikipedia.org/wiki/Gateway_(telecommunications)|Gateway]]
 +    * [[https://en.wikipedia.org/wiki/Firewall_(computing)|Firewall]]
 +    * [[https://en.wikipedia.org/wiki/Network_address_translation|NAT]]
 +  * Unix: users, groups, [[https://en.wikipedia.org/wiki/Access_control_list|ACL]]
 +  * SSH: key authentication, tunneling, SCP
 +
 +
 +
 +
 +===== Concepts =====
 +
 +
 +  * [[https://martinfowler.com/bliki/DataLake.html|Martin Fowler on Data Lake]]
 +  * [[https://knowledgent.com/whitepaper/design-successful-data-lake/|How to Design a Successful Data Lake]]
 +  * [[https://tidyfirst.substack.com/p/eventual-business-consistency|Eventual Business Consistency (Kent Beck)]]
  
  
Line 6: Line 46:
  
  
-https://hadoopecosystemtable.github.io/+  * [[http://mattturck.com/bigdata2017/|Big Data Landscape]], [[http://mattturck.com/wp-content/uploads/2017/05/Matt-Turck-FirstMark-2017-Big-Data-Landscape.png|Image]] 
 +  * [[https://wiki.apache.org/hadoop/Distributions%20and%20Commercial%20Support|Distributions]] 
 +  * [[https://hadoopecosystemtable.github.io/|Eco system]] 
 +  * [[https://www.cncf.io/|Cloud Native Computing Foundation]], Cloud Native Landscape 
 +    * [[https://landscape.cncf.io/|Interactive]] 
 +    * [[https://github.com/cncf/landscape|Static]] 
 +  * [[https://landscape.cd.foundation/|Continuous Delivery Landscape]] 
 +  * [[https://www.thoughtworks.com/radar|Thoughtworks Technology Radar]] 
 + 
 + 
 + 
 +===== Technologies ===== 
 + 
 + 
 +  * Apache Hadoop 
 +    * Apache HDFS 
 +    * Apache MapReduce 
 +    * Apache Parquet 
 +    * Apache Sentry 
 +  * [[https://oozie.apache.org/|Apache Oozie]]: [[https://en.wikipedia.org/wiki/Apache_Oozie|Worklow scheduling system]] 
 +  * Apache Sqoop 
 +  * Apache Hive 
 +  * Cloudera Impala 
 +  * Cloudera Hue 
 +  * Apache Spark with Python, Scala, R 
 +  * Apache Kafka 
 +  * Prometheus 
 +  * Grafana 
 +  * ELK stack 
 +    * Elasticsearch 
 +    * Logstash 
 +    * Kibana 
 +  * Apache Flink 
 +  * Apache Storm 
 +  * Apache Beam 
 +  * [[https://superset.apache.org/|Apache Superset]] 
 +  * [[https://trino.io/|Trino]]
  
  
Line 13: Line 89:
  
  
-Hadoop: https://wiki.apache.org/hadoop/HowToSetupYourDevelopmentEnvironment+  * [[https://12factor.net/|The Twelve-Factor App]] 
 +  * Hadoop: [[https://wiki.apache.org/hadoop/HowToSetupYourDevelopmentEnvironment|How to set up your development environment]] 
 +  * Spark 
 +    * [[http://spark.apache.org/docs/latest/spark-standalone.html#installing-spark-standalone-to-a-cluster|Installing Spark standalone to a cluster]] 
 +    * [[https://spark.apache.org/developer-tools.html|Spark developer tools]] 
 + 
 + 
 +===== Open Data ===== 
 + 
 + 
 +  * [[https://atlas.ripe.net/docs/result-streaming/|RIPE Atlas Result Streams]]
  
-Spark: 
  
-  * http://spark.apache.org/docs/latest/spark-standalone.html#installing-spark-standalone-to-a-cluster +===== Miscellaneous =====
-  * https://spark.apache.org/developer-tools.html+
  
 +  * [[https://seattledataguy.substack.com/p/normalization-vs-denormalization|Normalization vs. Denormalization – Seattle Data Guy]]
 +  * [[https://www.ibm.com/think/topics/semantic-layer|Semantic Layer]]
  
big-data-analytics.1511255588.txt.gz · Last modified: (external edit)