<?xml version="1.0" encoding="utf-8" ?><rss version="2.0"><channel><title><![CDATA[L, there!]]></title><description><![CDATA[只有当后悔取代了梦想，那才是真的老了。]]></description><link>https://blog.csdn.net/lingeio</link><language>zh-cn</language><generator>https://blog.csdn.net/</generator><copyright><![CDATA[Copyright &copy; lingeio]]></copyright><item><title><![CDATA[Apache大数据相关组件部署]]></title><link>https://blog.csdn.net/lingeio/article/details/124977031</link><guid>https://blog.csdn.net/lingeio/article/details/124977031</guid><author>lingeio</author><pubDate>Thu, 26 May 2022 01:00:01 +0800</pubDate><description><![CDATA[组件总览

	Zookeeper部署

		
			创建目录
				解压
				环境变量
				配置服务器编号
				配置
				同步
				启动
			
		
		Hadoop部署
		
			解压
				环境变量
				创建目录
				集群配置
				core-site.xml
					hdfs-site.xml
					mapred-site.xml
					yarn-site.xml
					workers
					启动
					初始化zkfc
						启动集..]]></description><category></category></item><item><title><![CDATA[大数据平台基础环境准备]]></title><link>https://blog.csdn.net/lingeio/article/details/124976977</link><guid>https://blog.csdn.net/lingeio/article/details/124976977</guid><author>lingeio</author><pubDate>Thu, 26 May 2022 00:45:15 +0800</pubDate><description><![CDATA[之前已经记录过，这里再记录一下，相以便快速使用。




安装依赖


sudo yum install -y epel-release psmisc nc net-tools rsync vim lrzsz ntp libzstd openssl-static tree iotop git



配置用户和权限

添加新用户 hdfs，并赋予 root 权限。

如果使用 hdfs 用户作为 Hadoop 默认用户，需要将 hdfs 用户也配置免密登录。


adduser hdfs
passwd hd.]]></description><category></category></item><item><title><![CDATA[Mac M1 VM Centos7 大数据数据湖测试]]></title><link>https://blog.csdn.net/lingeio/article/details/124976425</link><guid>https://blog.csdn.net/lingeio/article/details/124976425</guid><author>lingeio</author><pubDate>Thu, 26 May 2022 00:26:44 +0800</pubDate><description><![CDATA[Mac M1 通过VMan安装Centos7.9，并搭建 Hadoop/Hive/Kafka/Flink/Iceberg 本地进行数据湖测试。


问题：

Paralles Desktop 没找到免费的，所以用了VM，VM也可以网上找。

Centos7.9官方版本在VM中不成功，所以使用了别人编译的版本：在m1芯片的MacBook上安装centos7

JDK使用 yum 安装 arm64架构的1.8.322版本。

MySQL使用官网下载arm64版本。

大数据相关组件使用官网二进制包。



集.]]></description><category></category></item><item><title><![CDATA[Spark3 读写 S3 Parquet, Hive, Hudi]]></title><link>https://blog.csdn.net/lingeio/article/details/124817817</link><guid>https://blog.csdn.net/lingeio/article/details/124817817</guid><author>lingeio</author><pubDate>Tue, 17 May 2022 11:56:09 +0800</pubDate><description><![CDATA[Spark 读 S3 Parquet 写入 Hudi 表



目录

Spark 读 S3 Parquet 写入 Hudi 表



参考

关于S3，S3N和S3A的区别与联系

Spark 读写 S3 Parquet 文件

测试代码

pom.xml

配置文件

EMR Spark任务提交

spark-shell

spark-submit

Spark 读写 Hudi

本地测试

代码

集群上测试

spark-shell

spark-sql

Spark-submit

Hive 中测]]></description><category></category></item><item><title><![CDATA[近期关注的大数据开源项目]]></title><link>https://blog.csdn.net/lingeio/article/details/122820930</link><guid>https://blog.csdn.net/lingeio/article/details/122820930</guid><author>lingeio</author><pubDate>Tue, 08 Feb 2022 11:40:33 +0800</pubDate><description><![CDATA[截止至 2022-02-08




计算引擎

Service
			Git Star
			Contributors
			Release
			License
		Apache Flink
			18.1k
			1005
			v1.14.3
			Apache-2.0
		Apache Flink：https://github.com/apache/flink



数据开发

Service
			Git Star
			Contributors
			Release
			License
	.]]></description><category></category></item><item><title><![CDATA[Flink Hudi 测试]]></title><link>https://blog.csdn.net/lingeio/article/details/120565869</link><guid>https://blog.csdn.net/lingeio/article/details/120565869</guid><author>lingeio</author><pubDate>Thu, 30 Sep 2021 14:33:16 +0800</pubDate><description><![CDATA[组件版本


			组件
			
			
			版本
			
		
			Java
			
			
			1.8.251
			
		
			Scala
			
			
			1.12.14
			
		
			Flink
			
			
			1.12.2
			
		
			hudi
			
			
			0.9.0
			
		
			Hadoop
			
			
			2.9.2
			
		
			Hive
			
			
			2.3.6
			
		


将hdfs-site.xml，core]]></description><category></category></item><item><title><![CDATA[Flink Iceberg 测试]]></title><link>https://blog.csdn.net/lingeio/article/details/120505596</link><guid>https://blog.csdn.net/lingeio/article/details/120505596</guid><author>lingeio</author><pubDate>Mon, 27 Sep 2021 12:52:30 +0800</pubDate><description><![CDATA[本地搭建单节点Hadoop，根据官网示例进行Iceberg功能测试。


组件版本


			组件
			
			
			版本
			
		
			Java
			
			
			1.8.251
			
		
			Scala
			
			
			1.12.14
			
		
			Flink
			
			
			1.12.5
			
		
			Iceberg
			
			
			0.12.0
			
		
			Hadoop
			
			
			2.9.2
			
		
			Hive
			.]]></description><category></category></item><item><title><![CDATA[Flink 自定义UDTF]]></title><link>https://blog.csdn.net/lingeio/article/details/119599245</link><guid>https://blog.csdn.net/lingeio/article/details/119599245</guid><author>lingeio</author><pubDate>Wed, 11 Aug 2021 11:16:28 +0800</pubDate><description><![CDATA[Flink自定义函数实现列传行，数据格式为Json数据：[{"key1":"value1","key2":"value2"...}]


Java


@FunctionHint(output = @DataTypeHint("ROW&lt;drugUniversalName string, specifications string, goodsUnit string, " +
        "location string, instruction string, consumption strin.]]></description><category></category></item><item><title><![CDATA[Flink run application 提交 job]]></title><link>https://blog.csdn.net/lingeio/article/details/113888021</link><guid>https://blog.csdn.net/lingeio/article/details/113888021</guid><author>lingeio</author><pubDate>Sat, 20 Feb 2021 17:54:59 +0800</pubDate><description><![CDATA[1. 上传 flink 相关 plugins 到hdfs



2. 上传 flink 相关 依赖到 hdfs



3. 上传用户 jar 到 hdfs

4. 提交任务


flink run-application -t yarn-application \
-c com.sm.analysis.rdw.SdkDataEtlOdsToDwd \
-Djobmanager.memory.process.size=1600m \
-Dtaskmanager.memory.process.size=2048]]></description><category></category></item><item><title><![CDATA[Scala md5/sha256加密工具类]]></title><link>https://blog.csdn.net/lingeio/article/details/113666762</link><guid>https://blog.csdn.net/lingeio/article/details/113666762</guid><author>lingeio</author><pubDate>Thu, 04 Feb 2021 23:33:15 +0800</pubDate><description><![CDATA[import java.security.MessageDigest

/**
  * md5, sha256加密
  *
  * create by LiuJinHe 2020/3/27
  */
object EncodeUtils {

  def MD5Encode(input: String): String = {
    // 指定MD5加密算法
    val md5 = MessageDigest.getInstance("MD5")
    // 对输入数据进行加密,过程是先将字符串中.]]></description><category></category></item><item><title><![CDATA[Scala Druid连接池工具类]]></title><link>https://blog.csdn.net/lingeio/article/details/113666670</link><guid>https://blog.csdn.net/lingeio/article/details/113666670</guid><author>lingeio</author><pubDate>Thu, 04 Feb 2021 23:28:14 +0800</pubDate><description><![CDATA[package com.sm.utils

import java.sql.{Connection, PreparedStatement, SQLException, Statement}
import java.util.Properties

import com.alibaba.druid.pool.DruidDataSourceFactory
import com.sm.common.conf.ConfigManager
import com.sm.constants.Constants
impo.]]></description><category></category></item><item><title><![CDATA[Scala HBaseUtils]]></title><link>https://blog.csdn.net/lingeio/article/details/113666625</link><guid>https://blog.csdn.net/lingeio/article/details/113666625</guid><author>lingeio</author><pubDate>Thu, 04 Feb 2021 23:22:36 +0800</pubDate><description><![CDATA[package com.sm.utils

import java.util.Properties
import java.util.concurrent.{ExecutorService, Executors}

import com.sm.common.conf.ConfigManager
import com.sm.constants.Constants
import org.apache.flink.hbase.shaded.org.apache.hadoop.hbase.{HBaseConfig.]]></description><category></category></item><item><title><![CDATA[Scala GuavaUtils]]></title><link>https://blog.csdn.net/lingeio/article/details/113666575</link><guid>https://blog.csdn.net/lingeio/article/details/113666575</guid><author>lingeio</author><pubDate>Thu, 04 Feb 2021 23:19:23 +0800</pubDate><description><![CDATA[package com.sm.utils

import java.util.concurrent.TimeUnit

import com.sm.utils.cache.CacheService
import org.apache.flink.shaded.guava18.com.google.common.cache.{Cache, CacheBuilder, CacheLoader}
import org.apache.flink.shaded.guava18.com.google.common.u.]]></description><category></category></item><item><title><![CDATA[Scala json4s Json解析工具类]]></title><link>https://blog.csdn.net/lingeio/article/details/113666332</link><guid>https://blog.csdn.net/lingeio/article/details/113666332</guid><author>lingeio</author><pubDate>Thu, 04 Feb 2021 23:02:10 +0800</pubDate><description><![CDATA[package com.sm.utils

import java.util

import com.sm.common.utils.DateUtil
import org.apache.commons.lang3.StringUtils
import org.json4s._
import org.json4s.jackson.JsonMethods._
import org.json4s.jackson.Serialization
import org.json4s.jackson.Serializa.]]></description><category></category></item><item><title><![CDATA[FlinkSQL 数据去重，读写HBase，Kafka]]></title><link>https://blog.csdn.net/lingeio/article/details/109225893</link><guid>https://blog.csdn.net/lingeio/article/details/109225893</guid><author>lingeio</author><pubDate>Thu, 22 Oct 2020 17:54:05 +0800</pubDate><description><![CDATA[需求：日活明细分析，需要保存日活明细数据。通过对登录数据去重得到日活，但是因为乱序数据，客户端时间可能会存在变化。


1. Flink窗口排序去重，写入HBase。


import java.time.Duration

import com.sm.common.conf.PropManager
import com.sm.constants.Constants
import com.sm.utils.FlinkUtils
import org.apache.flink.api.common.res..]]></description><category></category></item><item><title><![CDATA[FlinkSQL Kafka to Hive]]></title><link>https://blog.csdn.net/lingeio/article/details/109224905</link><guid>https://blog.csdn.net/lingeio/article/details/109224905</guid><author>lingeio</author><pubDate>Thu, 22 Oct 2020 16:36:32 +0800</pubDate><description><![CDATA[import java.time.Duration

import com.sm.common.conf.PropManager
import com.sm.constants.Constants
import com.sm.utils.FlinkUtils
import org.apache.flink.streaming.api.{CheckpointingMode, TimeCharacteristic}
import org.apache.flink.streaming.api.environme.]]></description><category></category></item><item><title><![CDATA[FlinkSQL MySQL CDC]]></title><link>https://blog.csdn.net/lingeio/article/details/109224123</link><guid>https://blog.csdn.net/lingeio/article/details/109224123</guid><author>lingeio</author><pubDate>Thu, 22 Oct 2020 16:20:53 +0800</pubDate><description><![CDATA[package com.sm.job

import com.sm.function.udf.{ParseDeviceCode, ParsePopularize}
import com.sm.utils.FlinkUtils
import org.apache.flink.streaming.api.scala._
import org.apache.flink.table.api.bridge.scala.StreamTableEnvironment
import org.apache.log4j...]]></description><category></category></item><item><title><![CDATA[FlinkSQL 读写 MySQL]]></title><link>https://blog.csdn.net/lingeio/article/details/109223441</link><guid>https://blog.csdn.net/lingeio/article/details/109223441</guid><author>lingeio</author><pubDate>Thu, 22 Oct 2020 15:41:32 +0800</pubDate><description><![CDATA[package com.sm.job

import com.sm.utils.FlinkUtils
import org.apache.flink.streaming.api.scala._
import org.apache.flink.table.api.bridge.scala.StreamTableEnvironment
import org.apache.log4j.Level
import org.slf4j.LoggerFactory

/**
  * flinkSQL 读写 mysql
.]]></description><category></category></item><item><title><![CDATA[FlinkSQL MySQL维表关联]]></title><link>https://blog.csdn.net/lingeio/article/details/109221881</link><guid>https://blog.csdn.net/lingeio/article/details/109221881</guid><author>lingeio</author><pubDate>Thu, 22 Oct 2020 15:25:33 +0800</pubDate><description><![CDATA[需求：消费Kafka数据，进行数据清洗及维表关联补齐字段，最后结果写入Kafka。



import java.time.Duration

import com.sm.function.udf._
import com.sm.common.conf.PropManager
import com.sm.constants.Constants
import com.sm.utils.FlinkUtils
import org.apache.flink.api.common.restartstrategy..]]></description><category></category></item><item><title><![CDATA[FlinkSQL Kafka to Kafka]]></title><link>https://blog.csdn.net/lingeio/article/details/107969282</link><guid>https://blog.csdn.net/lingeio/article/details/107969282</guid><author>lingeio</author><pubDate>Wed, 12 Aug 2020 23:39:34 +0800</pubDate><description><![CDATA[FlinkSql 1.11 读写 Kafka 简单测试。


FlinkSql 读 Kafka


import org.apache.flink.streaming.api.scala._
import org.apache.flink.table.api.EnvironmentSettings
import org.apache.flink.table.api.bridge.scala.StreamTableEnvironment
import org.apache.flink.types.Row

.]]></description><category></category></item></channel></rss>