0%

Centos7上部署单机版hbase

下载安装包

hbase

hbase官网

解压

tar -zxvf hbase-2.2.4.bin.tar.gz

配置hbase-env.sh

部署JDK,查看JAVA_HOME路径。

[root@node1 hbase]# echo $JAVA_HOME
/usr/local/bin/jdk1.8.0_112

配置conf/hbase-env.sh

...
# 配置java路径
export JAVA_HOME=/usr/local/bin/jdk1.8.0_112
# 配置是否使用自身Zookeeper
export HBASE_MANAGES_ZK=true
...

编辑conf/hbase-site.xml

<configuration>
    <property>
        <name>hbase.rootdir</name>
        <value>file:///root/hbase/rootdir</value>
        <description>The directory shared byregion servers.</description>
    </property>
    <property>
        <name>fs.defaultFS</name>
&nbsp;       <value>file:///root/hbase/dfs</value>
    </property>
    <!-- hbase的端口 -->
    <!-- <property>
        <name>hbase.zookeeper.property.clientPort</name>
        <value>2181</value>
        <description>Property from ZooKeeper'sconfig zoo.cfg. The port at which the clients will connect.
        </description>
    </property> -->
    <!--&nbsp; 超时时间 -->
    <!-- <property>
        <name>zookeeper.session.timeout</name>
        <value>120000</value>
    </property> -->
    <!--&nbsp; zookeeper 集群配置。如果是集群,则添加其它的主机地址 -->
    <!-- <property>
        <name>hbase.zookeeper.quorum</name>
        <value>localhost:2181</value>
    </property> -->
    <property>
        <name>hbase.tmp.dir</name>
        <value>/root/hbase/tmp</value>
    </property>
    <!-- <property>
        <name>hbase.cluster.distributed</name>
        <value>false</value>
    </property> -->
    <property>
        <name>hbase.zookeeper.property.dataDir</name>
        <value>/root/hbase/datadir</value>
    </property>
    <property>
        <name>zookeeper.znode.parent</name>
        <value>/hbase</value>
    </property>
</configuration>

启动

./bin/start-hbase.sh

Web UI

http://192.168.41.128:16010

进入shell命令行

./bin/hbase shell

简单测试

创建一个表,需要指明table nameColumnFamily name

hbase(main):002:0> create 'test', 'cf'
Created table test
Took 0.8251 seconds
=> Hbase::Table - test

确认

hbase(main):004:0> list 'test'
TABLE
test
1 row(s)
Took 0.0077 seconds
=> ["test"]

describe ‘test’

hbase(main):005:0> describe 'test'
Table test is ENABLED
test
COLUMN FAMILIES DESCRIPTION
{NAME => 'cf', VERSIONS => '1', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPL
ICATION_SCOPE => '0', BLOOMFILTER => 'ROW', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'false', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} 

1 row(s)

QUOTAS                                                                                                                                                                  0 row(s)
Took 0.2132 seconds 

写入数据

hbase(main):003:0> put 'test', 'row1', 'cf:a', 'value1'
0 row(s) in 0.0850 seconds

hbase(main):004:0> put 'test', 'row2', 'cf:b', 'value2'
0 row(s) in 0.0110 seconds

hbase(main):005:0> put 'test', 'row3', 'cf:c', 'value3'
0 row(s) in 0.0100 seconds

查看写入的数据

hbase(main):006:0> scan 'test'
ROW                                      COLUMN+CELL
 row1                                    column=cf:a, timestamp=1586504678931, value=value1
 row2                                    column=cf:b, timestamp=1586504712029, value=value2
 row3                                    column=cf:c, timestamp=1586504717104, value=value3
3 row(s) in 0.0230 seconds

获取一行数据

hbase(main):007:0> get 'test', 'row1'
COLUMN                                   CELL
 cf:a                                    timestamp=1586504678931, value=value1
1 row(s) in 0.0350 seconds

获取一个字段

hbase(main):007:0> get 'test', 'row1','cf:a'
COLUMN                                   CELL
 cf:a                                    timestamp=1586504678931, value=value1
1 row(s) in 0.0350 seconds

返回一个字段的多个版本值

默认Hbase列族最多保存一个版本的数据,可以通过下面命令修改,也可以使用HColumnDescriptor

alter 'test', NAME => 'cf', VERSIONS => 3

HBase 0.98.2开始,您可以通过在hbase-site.xml中设置hbase.column.max.version为所有新创建列保留的最大版本数指定一个全局默认值。

get 'test', 'row1', {COLUMN=>'cf:a',VERSIONS=>3}

禁用或者恢复表

hbase(main):008:0> disable 'test'
0 row(s) in 1.1820 seconds

hbase(main):009:0> enable 'test'
0 row(s) in 0.1770 seconds

添加一个列族

hbase(main):012:0> alter 'test',{NAME=>'haha'}
Updating all regions with the new schema...
1/1 regions updated.
Done.
Took 2.0149 seconds 
hbase(main):014:0> alter 'test',NAME=>'hehe'
Updating all regions with the new schema...
1/1 regions updated.
Done.
Took 1.9555 seconds 

删除一个列族

hbase(main):009:0> alter 'test',{NAME=>'haha',METHOD=>'delete'}
Updating all regions with the new schema...
1/1 regions updated.
Done.
Took 1.9296 seconds
hbase(main):013:0> alter 'test','delete'=>'haha'
Updating all regions with the new schema...
1/1 regions updated.
Done.
Took 1.8706 seconds 
# 添加一个列族,同时删除一个列族
hbase(main):011:0> alter 'test', {NAME => 'hehe'}, {NAME => 'myInfo', METHOD => 'delete'}
Updating all regions with the new schema...
1/1 regions updated.
Done.
Updating all regions with the new schema...
1/1 regions updated.
Done.
0 row(s) in 3.8260 seconds

清空表

hbase(main):013:0> truncate 'test'
Truncating 'test' table (it may take a while):
 - Disabling table...
 - Truncating table...
0 row(s) in 3.6760 seconds

删除表

删除表或者修改表的配置前,需要禁用表

hbase(main):011:0> drop 'test'
0 row(s) in 0.1370 seconds

退出Shell

quit 或者 Ctrl+D

停止Hbase

./bin/stop-hbase.sh

@Scheduled

实现cron表达式从application.yml文件中获取

dbc:
  cron: 0/5 * * * * ?
@Component
@EnableScheduling
public class CompareSchedule {

    @Scheduled(cron = "${dbc.cron}")
    public void schedule(){
        System.out.println(1);
    }

}

cron表达式含义

cron一共有7位,但是最后一位是年,可以留空,所以我们可以写6位:

  • 第一位,表示秒,取值0-59
  • 第二位,表示分,取值0-59 
  • 第三位,表示小时,取值0-23 
  • 第四位,日期天/日,取值1-31
  • 第五位,日期月份,取值1-12 
  • 第六位,星期,取值1-7,星期一,星期二…,注:不是第1周,第二周的意思 另外:1表示星期天,2表示星期一。
  • 第七位,年份,可以留空,取值1970-2099

 

cron中,还有一些特殊的符号,含义如下:

  • (*)星号:可以理解为每的意思,每秒,每分,每天,每月,每年… 
  • (?)问号:问号只能出现在日期和星期这两个位置,表示这个位置的值不确定,每天3点执行,所以第六位星期的位置,我们是不需要关注的,就是不确定的值。同时:日期和星期是两个相互排斥的元素,通过问号来表明不指定值。比如,1月10日,比如是星期1,如果在星期的位置是另指定星期二,就前后冲突矛盾了。
  • (-)减号:表达一个范围,如在小时字段中使用“10-12”,则表示从10到12点,即10,11,12 
  • (,)逗号:表达一个列表值,如在星期字段中使用“1,2,4”,则表示星期一,星期二,星期四 
  • (/)斜杠:如:x/y,x是开始值,y是步长,比如在第一位(秒) 0/15就是,从0秒开始,每15秒,最后就是0,15,30,45,60 另:*/y,等同于0/y

 

下面列举几个例子供大家来验证:

  • 0 0 3 * * ? 每天3点执行 
  • 0 5 3 * * ? 每天3点5分执行 
  • 0 5 3 ? * * 每天3点5分执行,与上面作用相同
  • 0 5/10 3 * * ? 每天3点的 5分,15分,25分,35分,45分,55分这几个时间点执行 
  • 0 10 3 ? * 1 每周星期天,3点10分 执行,注:1表示星期天 
  • 0 10 3 ? * 1 #3 每个月的第三个星期,星期天 执行,#号只能出现在星期的位置

 

方法1

这种方法能够查询到一些表信息,但是不够详细

PreparedStatement pst = connection.prepareStatement("select * from t_gateway_logs where 1=2");
ResultSet rs = pst.executeQuery();
ResultSetMetaData rsd = rs.getMetaData();
for(int i = 0; i < rsd.getColumnCount(); i++) {
    System.out.print("java类型:"+rsd.getColumnClassName(i + 1));
    System.out.print("  数据库类型:"+rsd.getColumnTypeName(i + 1));
    System.out.print("  字段名称:"+rsd.getColumnName(i + 1));
    System.out.print("  字段长度:"+rsd.getColumnDisplaySize(i + 1));
    System.out.print("  是否为空:"+rsd.isNullable(i + 1));
    System.out.print("  是否自增:"+rsd.isAutoIncrement(i + 1));
    System.out.println();
}

方法二

查询出来的内容非常多。

查询出库中所有表

ResultSet tableRet = connection.getMetaData().getTables(null, "%","%",new String[]{"TABLE"});
while(tableRet.next()) {
    System.out.println(tableRet.getString("TABLE_NAME"));
}

查询某个表的主键信息

ResultSet pkInfo = connection.getMetaData().getPrimaryKeys(null, "%", "test_table");
while (pkInfo.next()) {
    System.out.println("TABLE_CAT:" + pkInfo.getString("TABLE_CAT") + " ");
    System.out.println("TABLE_NAME:" + pkInfo.getString("TABLE_NAME") + " ");
    System.out.println("COLUMN_NAME:" + pkInfo.getString("COLUMN_NAME") + " ");
    System.out.println("PK_NAME:" + pkInfo.getString("PK_NAME") + " ");
    System.out.println("TABLE_SCHEM:" + pkInfo.getString("TABLE_SCHEM") + " ");
    System.out.println("KEY_SEQ:" + pkInfo.getString("KEY_SEQ") + " ");
}

查询某个表的详细信息

ResultSet colRet = connection.getMetaData().getColumns(null, "%", "test_table", "%");
while (colRet.next()) {
    System.out.println(colRet.getString("TABLE_CAT"));
    System.out.println(colRet.getString("TABLE_SCHEM"));
    System.out.println(colRet.getString("TABLE_NAME"));
    System.out.println(colRet.getString("COLUMN_NAME"));
    System.out.println(colRet.getString("DATA_TYPE"));
    System.out.println(colRet.getString("TYPE_NAME"));
    System.out.println(colRet.getInt("COLUMN_SIZE"));
    System.out.println(colRet.getInt("BUFFER_LENGTH"));
    System.out.println(colRet.getInt("DECIMAL_DIGITS"));
    System.out.println(colRet.getInt("NUM_PREC_RADIX"));
    System.out.println(colRet.getInt("NULLABLE"));
    System.out.println(colRet.getString("REMARKS"));
    System.out.println(colRet.getString("COLUMN_DEF"));
    System.out.println(colRet.getInt("SQL_DATA_TYPE"));
    System.out.println(colRet.getInt("SQL_DATETIME_SUB"));
    System.out.println(colRet.getInt("CHAR_OCTET_LENGTH"));
    System.out.println(colRet.getInt("ORDINAL_POSITION"));
    System.out.println(colRet.getBoolean("IS_NULLABLE"));
    System.out.println(colRet.getInt("SCOPE_CATALOG"));
    System.out.println(colRet.getInt("SCOPE_SCHEMA"));
    System.out.println(colRet.getInt("SCOPE_TABLE"));
    System.out.println(colRet.getInt("SOURCE_DATA_TYPE"));
    System.out.println(colRet.getBoolean("IS_AUTOINCREMENT"));
    System.out.println(colRet.getBoolean("IS_GENERATEDCOLUMN"));
}

问题描述

Maven打包jar用的默认插件是maven-jar-plugin,默认Maven生成的JAR包只包含了编译生成的.class文件和项目资源文件。有两个问题:

  • 运行时提示没有主清单属性;
  • 依赖的jar包也没有打包进去.

解决办法

maven打包插件有很多种:

  • maven-shade-plugin
  • maven-assembly-plugin
  • maven-jar-plugin

我们直接重新定义maven-jar-plugin,详细配置如下:

<build>
    <plugins>
        <plugin>
            <groupId>org.apache.maven.plugins</groupId>
            <artifactId>maven-shade-plugin</artifactId>
            <version>1.2.1</version>
            <executions>
                <execution>
                    <phase>package</phase>
                    <goals>
                        <goal>shade</goal>
                    </goals>
                    <configuration>
                        <transformers>
                            <transformer
                                    implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
                                <!-- 这里指定main函数 -->
                                <mainClass>com.deri.App</mainClass>
                            </transformer>
                        </transformers>
                    </configuration>
                </execution>
            </executions>
        </plugin>
    </plugins>
</build>

问题描述

使用logback打印日志,启动服务时logback会打印一些自身的调试信息如下,会打印它检查了哪些配置、看见了哪些配置:

18:00:22,729 |-INFO in ch.qos.logback.classic.LoggerContext[default] - Could NOT find resource [logback-test.xml]
18:00:22,729 |-INFO in ch.qos.logback.classic.LoggerContext[default] - Could NOT find resource [logback.groovy]
18:00:22,729 |-INFO in ch.qos.logback.classic.LoggerContext[default] - Found resource [logback.xml] at [file:/D:/DeriGateway/DBCompare/target/classes/logback.xml]
18:00:22,770 |-INFO in ch.qos.logback.classic.joran.action.ConfigurationAction - debug attribute not set
18:00:22,775 |-INFO in ch.qos.logback.core.joran.action.TimestampAction - Using current interpretation time, i.e. now, as time reference.
18:00:22,788 |-INFO in ch.qos.logback.core.joran.action.TimestampAction - Adding property to the context with key="DATETIME" and value="2020-04-01 18:00:22" to the LOCAL scope
18:00:22,788 |-INFO in ch.qos.logback.core.joran.action.AppenderAction - About to instantiate appender of type [ch.qos.logback.core.ConsoleAppender]
18:00:22,793 |-INFO in ch.qos.logback.core.joran.action.AppenderAction - Naming appender as [STDOUT]
18:00:22,796 |-INFO in ch.qos.logback.core.joran.action.NestedComplexPropertyIA - Assuming default type [ch.qos.logback.classic.encoder.PatternLayoutEncoder] for [encoder] property
18:00:22,813 |-INFO in ch.qos.logback.core.joran.action.AppenderAction - About to instantiate appender of type [ch.qos.logback.core.rolling.RollingFileAppender]
18:00:22,816 |-INFO in ch.qos.logback.core.joran.action.AppenderAction - Naming appender as [FILEOUT]
18:00:22,817 |-INFO in ch.qos.logback.core.joran.action.NestedComplexPropertyIA - Assuming default type [ch.qos.logback.classic.encoder.PatternLayoutEncoder] for [encoder] property
18:00:22,828 |-INFO in c.q.l.core.rolling.TimeBasedRollingPolicy@984849465 - No compression will be used
18:00:22,829 |-INFO in c.q.l.core.rolling.TimeBasedRollingPolicy@984849465 - Will use the pattern ./logs//log.%d.%i.log for the active file
18:00:22,830 |-INFO in ch.qos.logback.core.rolling.SizeAndTimeBasedFNATP@2eee9593 - The date pattern is 'yyyy-MM-dd' from file name pattern './logs//log.%d.%i.log'.
18:00:22,830 |-INFO in ch.qos.logback.core.rolling.SizeAndTimeBasedFNATP@2eee9593 - Roll-over at midnight.
18:00:22,832 |-INFO in ch.qos.logback.core.rolling.SizeAndTimeBasedFNATP@2eee9593 - Setting initial period to Wed Apr 01 18:00:22 CST 2020
18:00:22,833 |-WARN in ch.qos.logback.core.rolling.SizeAndTimeBasedFNATP@2eee9593 - SizeAndTimeBasedFNATP is deprecated. Use SizeAndTimeBasedRollingPolicy instead
18:00:22,833 |-WARN in ch.qos.logback.core.rolling.SizeAndTimeBasedFNATP@2eee9593 - For more information see http://logback.qos.ch/manual/appenders.html#SizeAndTimeBasedRollingPolicy
18:00:22,834 |-INFO in ch.qos.logback.core.rolling.RollingFileAppender[FILEOUT] - Active log file name: ./logs//log.2020-04-01.0.log
18:00:22,834 |-INFO in ch.qos.logback.core.rolling.RollingFileAppender[FILEOUT] - File property is set to [null]
18:00:22,835 |-INFO in ch.qos.logback.classic.joran.action.RootLoggerAction - Setting level of ROOT logger to INFO
18:00:22,835 |-INFO in ch.qos.logback.core.joran.action.AppenderRefAction - Attaching appender named [STDOUT] to Logger[ROOT]
18:00:22,835 |-INFO in ch.qos.logback.core.joran.action.AppenderRefAction - Attaching appender named [FILEOUT] to Logger[ROOT]
18:00:22,835 |-INFO in ch.qos.logback.classic.joran.action.ConfigurationAction - End of configuration.
18:00:22,836 |-INFO in ch.qos.logback.classic.joran.JoranConfigurator@7907ec20 - Registering current configuration as safe fallback point

关闭logback自身调试信息打印

只需增加一行以下配置即可。将状态信息监听器设置为无操作监听器。

<configuration>
    <statusListener class="ch.qos.logback.core.status.NopStatusListener" />
</configuration>

完整logback.xml

<?xml version="1.0" encoding="UTF-8"?>
<configuration>
    <statusListener class="ch.qos.logback.core.status.NopStatusListener" />
    <property name="ROOT" value="./logs/" />
    <property name="FILESIZE" value="100MB" />
    <property name="MAXHISTORY" value="30" />
    <timestamp key="DATETIME" datePattern="yyyy-MM-dd HH:mm:ss" />
    <!-- 控制台打印 -->
    <appender name="STDOUT" class="ch.qos.logback.core.ConsoleAppender">
        <encoder>
            <pattern>[%-5level] %d{${DATETIME}} [%thread] %logger{36} - %m%n</pattern>
        </encoder>
    </appender>
    <!-- 输入到文件,按日期和文件大小 -->
    <appender name="FILEOUT" class="ch.qos.logback.core.rolling.RollingFileAppender">
        <encoder>
            <pattern>[%-5level] %d{${DATETIME}} [%thread] %logger{36} - %m%n</pattern>
        </encoder>
        <rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">
            <fileNamePattern>${ROOT}/log.%d.%i.log</fileNamePattern>
            <maxHistory>${MAXHISTORY}</maxHistory>
            <timeBasedFileNamingAndTriggeringPolicy class="ch.qos.logback.core.rolling.SizeAndTimeBasedFNATP">
                <maxFileSize>${FILESIZE}</maxFileSize>
            </timeBasedFileNamingAndTriggeringPolicy>
        </rollingPolicy>
    </appender>
    <!-- Logger 根目录 -->
    <root level="INFO">
        <appender-ref ref="STDOUT" />
        <appender-ref ref="FILEOUT" />
    </root>
</configuration>