lrnz.name

User Tools

Site Tools

Trace: • dokuwiki • playground • python • big-data-analytics • ssh • software • development
big-data-analytics

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revisionPrevious revision
Next revision		Previous revision
big-data-analytics [2018-01-18 10:54] – mrksbig-data-analytics [2026-04-22 10:54] (current) – mrks
Line 1: Line 1:
 ====== Big Data Analytics ====== ====== Big Data Analytics ======
 +
 +
 +
 +===== Fundamentals =====
 +
 +
 +  * [[https://en.wikipedia.org/wiki/Character_encoding|Character encodings]]
 +    * [[https://en.wikipedia.org/wiki/US-ASCII|ASCII]]
 +    * [[https://en.wikipedia.org/wiki/ISO/IEC_8859-1|ISO 8859-1]]
 +    * [[https://en.wikipedia.org/wiki/Unicode|Unicode]]
 +    * [[https://en.wikipedia.org/wiki/UTF-8|UTF-8]]
 +    * [[https://en.wikipedia.org/wiki/UTF-16|UTF-16]]
 +    * [[https://en.wikipedia.org/wiki/UTF-32|UTF-32]]
 +    * [[https://en.wikipedia.org/wiki/EBCDIC|EBCDIC]]
 +  * [[https://en.wikipedia.org/wiki/File_format|File formats]]
 +    * [[https://en.wikipedia.org/wiki/Comma-separated_values|CSV]]
 +    * [[https://en.wikipedia.org/wiki/XML|XML]] and [[https://en.wikipedia.org/wiki/XML_Schema_(W3C)|XSD]]
 +    * [[https://en.wikipedia.org/wiki/JSON|JSON]]
 +  * [[https://en.wikipedia.org/wiki/Database|Databases]]
 +    * [[https://en.wikipedia.org/wiki/SQL|SQL]]
 +    * [[https://en.wikipedia.org/wiki/Java_Database_Connectivity|JDBC]]
 +  * [[https://en.wikipedia.org/wiki/Computer_network|Network]]
 +    * [[https://en.wikipedia.org/wiki/IPv4|IPv4]] and [[https://en.wikipedia.org/wiki/Subnetwork|network mask]]
 +    * [[https://en.wikipedia.org/wiki/IPv6|IPv6]]
 +    * [[https://en.wikipedia.org/wiki/IP_routing|Routing]] and [[https://en.wikipedia.org/wiki/Router_(computing)|Router]]
 +    * [[https://en.wikipedia.org/wiki/Gateway_(telecommunications)|Gateway]]
 +    * [[https://en.wikipedia.org/wiki/Firewall_(computing)|Firewall]]
 +    * [[https://en.wikipedia.org/wiki/Network_address_translation|NAT]]
 +  * Unix: users, groups, [[https://en.wikipedia.org/wiki/Access_control_list|ACL]]
 +  * SSH: key authentication, tunneling, SCP
 +
 +
 +
 +
 +===== Concepts =====
 +
 +
 +  * [[https://martinfowler.com/bliki/DataLake.html|Martin Fowler on Data Lake]]
 +  * [[https://knowledgent.com/whitepaper/design-successful-data-lake/|How to Design a Successful Data Lake]]
 +  * [[https://tidyfirst.substack.com/p/eventual-business-consistency|Eventual Business Consistency (Kent Beck)]]
  
  
Line 6: Line 46:
  
  
-  * Big Data Landscape +  * [[http://mattturck.com/bigdata2017/|Big Data Landscape]], [[http://mattturck.com/wp-content/uploads/2017/05/Matt-Turck-FirstMark-2017-Big-Data-Landscape.png|Image]] 
-    * http://mattturck.com/bigdata2017/ +  * [[https://wiki.apache.org/hadoop/Distributions%20and%20Commercial%20Support|Distributions]] 
-    * http://mattturck.com/wp-content/uploads/2017/05/Matt-Turck-FirstMark-2017-Big-Data-Landscape.png +  * [[https://hadoopecosystemtable.github.io/|Eco system]] 
-  * Distributions: https://wiki.apache.org/hadoop/Distributions%20and%20Commercial%20Support +  * [[https://www.cncf.io/|Cloud Native Computing Foundation]], Cloud Native Landscape 
-  * Eco system: https://hadoopecosystemtable.github.io/+    * [[https://landscape.cncf.io/|Interactive]] 
 +    * [[https://github.com/cncf/landscape|Static]] 
 +  * [[https://landscape.cd.foundation/|Continuous Delivery Landscape]] 
 +  * [[https://www.thoughtworks.com/radar|Thoughtworks Technology Radar]]
  
  
Line 18: Line 61:
  
   * Apache Hadoop   * Apache Hadoop
-    * HDFS +    * Apache HDFS 
-    * MapReduce +    * Apache MapReduce 
-    * Parquet +    * Apache Parquet 
-    * Sentry+    * Apache Sentry 
 +  * [[https://oozie.apache.org/|Apache Oozie]]: [[https://en.wikipedia.org/wiki/Apache_Oozie|Worklow scheduling system]]
   * Apache Sqoop   * Apache Sqoop
   * Apache Hive   * Apache Hive
Line 28: Line 72:
   * Apache Spark with Python, Scala, R   * Apache Spark with Python, Scala, R
   * Apache Kafka   * Apache Kafka
 +  * Prometheus
   * Grafana   * Grafana
   * ELK stack   * ELK stack
Line 36: Line 81:
   * Apache Storm   * Apache Storm
   * Apache Beam   * Apache Beam
 +  * [[https://superset.apache.org/|Apache Superset]]
 +  * [[https://trino.io/|Trino]]
  
  
Line 42: Line 89:
  
  
-Hadoop: https://wiki.apache.org/hadoop/HowToSetupYourDevelopmentEnvironment+  * [[https://12factor.net/|The Twelve-Factor App]] 
 +  * Hadoop: [[https://wiki.apache.org/hadoop/HowToSetupYourDevelopmentEnvironment|How to set up your development environment]] 
 +  * Spark 
 +    * [[http://spark.apache.org/docs/latest/spark-standalone.html#installing-spark-standalone-to-a-cluster|Installing Spark standalone to a cluster]] 
 +    * [[https://spark.apache.org/developer-tools.html|Spark developer tools]] 
 + 
 + 
 +===== Open Data ===== 
 + 
 + 
 +  * [[https://atlas.ripe.net/docs/result-streaming/|RIPE Atlas Result Streams]]
  
-Spark: 
  
-  * http://spark.apache.org/docs/latest/spark-standalone.html#installing-spark-standalone-to-a-cluster +===== Miscellaneous =====
-  * https://spark.apache.org/developer-tools.html+
  
 +  * [[https://seattledataguy.substack.com/p/normalization-vs-denormalization|Normalization vs. Denormalization – Seattle Data Guy]]
 +  * [[https://www.ibm.com/think/topics/semantic-layer|Semantic Layer]]
  
big-data-analytics.1516269254.txt.gz · Last modified: (external edit)