基于CDH的Solr+Hbase+KV构建二级索引
在CDH主界面，选择“添加服务”
选择“solr”，之后点击“继续”
2.安装Key-Value Store Indexer
与上面操作雷同
3.构建二级索引 3.1创建测试表 3.1.1创建测试表（phoenix）drop table "ZJS_ORDER"."ALADIN_WAYBILLSTATUS";
CREATE TABLE "ZJS_ORDER"."ALADIN_WAYBILLSTATUS"
(
ydh varchar NOT NULL PRIMARY KEY,
khbh varchar,
khdh varchar,
kdsj varchar,
dsf_data varchar,
zjz_data varchar,
kd_data varchar,
ps_data varchar,
qs_data varchar,
rk_data varchar,
ck_data varchar,
yc_data varchar,
zf_data varchar,
sr_data varchar,
sk_data varchar
)
SALT_BUCKETS = 9,
COMPRESSION='SNAPPY',
REPLICATION_SCOPE=1,
COLUMN_ENCODED_BYTES=0,
DEFAULT_COLUMN_FAMILY='0',
BLOOMFILTER='ROW';
3.1.2插入一条测试数据upsert into "ZJS_ORDER"."ALADIN_WAYBILLSTATUS"
(ydh,khbh,khdh,kdsj,dsf_data,zjz_data,kd_data,ps_data,
qs_data,rk_data,ck_data,yc_data,zf_data,sr_data,sk_data)
values('a01010100111','769520','1235664','2019-12-01 12:00:26',
'A','b','c','d','1','2','3','4','5','6','7'
);
select * from "ZJS_ORDER"."ALADIN_WAYBILLSTATUS";上述两步操作成功后，显示如下：
describe 'ZJS_ORDER.ALADIN_WAYBILLSTATUS'
scan 'ZJS_ORDER.ALADIN_WAYBILLSTATUS'
创建配置目录，可以自定义路径
mkdir -p /opt/cdhsolr
之后初始化默认配置文件
solrctl instancedir --generate /opt/cdhsolr/zjs_order_aladin_waybillstatus
3.2.2.1schema.xml
vi /opt/cdhsolr/zjs_order_aladin_waybillstatus/conf/schema.xml
编辑scheme.xml文件，把hbase表中需要索引的列添加到scheme.xml 文件的filed节点,其中的name属性值要与Morphline.conf文件中的outputField属性值对应
3.2.2.2 solrconfig.xml
vi /opt/cdhsolr/zjs_order_aladin_waybillstatus/conf/solrconfig.xml
打开自动提交
solrctl instancedir --create \
zjs_order_aladin_waybillstatus /opt/cdhsolr/zjs_order_aladin_waybillstatus
首先，来看一下创建之前的solr界面
执行创建命令：solrctl collection --create zjs_order_aladin_waybillstatus -s 3 –r 3 –m 9
（s表示设置Shard数为3，-r表示设置的replica数为3,-m表示最大shards数目9）
之后，我们可以在http://192.168.56.11:8983中看到刚刚创建的collection
vi /opt/cdhsolr/zjs_order_aladin_waybillstatus/morphline-hbase-mapper.xml
vi /opt/cdhsolr/zjs_order_aladin_waybillstatus/morphlines.conf
SOLR_LOCATOR :{
zkHost :"$ZK_HOST"
}
morphlines : [
{
id :zjs_order_aladin_waybillstatusMap
importCommands : ["org.kitesdk.**", "com.ngdata.**"]
commands : [
{
extractHBaseCells {
mappings : [
{
inputColumn : "0:YDH"
outputField : "ZJS_ORDER.ALADIN_WAYBILLSTATUS_0_YDH"
type : string
source : value
},
{
inputColumn : "0:KHBH"
outputField : "ZJS_ORDER.ALADIN_WAYBILLSTATUS_0_KHBH"
type : string
source : value
},
………………….
{
inputColumn : "0:SK_DATA"
outputField : "ZJS_ORDER.ALADIN_WAYBILLSTATUS_0_SK_DATA"
type : string
source : value
}
]
}
}
{ logDebug { format : "output record: {}", args : ["@{}"] } }
]
}
]
在CDH中配置Key-Value Store Indexer，并重启。
创建索引，将hbase与solr、zk关联起来
hbase-indexer add-indexer \
--name zjs_order_aladin_waybillstatusIndexer \
--indexer-conf /opt/cdhsolr/zjs_order_aladin_waybillstatus/morphline-hbase-mapper.xml \
--connection-param solr.zk=hbase1:2181,hbase2:2181,hbase3:2181/solr \
--connection-param solr.collection=zjs_order_aladin_waybillstatus \
--zookeeper hbase1:2181,hbase2:2181,hbase3:2181
hbase-indexer list-indexers
从界面上来看，solr里已有刚才创建的索引集合，此时，还未有相关的索引值信息
3.2.11再次插入测试数据并查看结果upsert into "ZJS_ORDER"."ALADIN_WAYBILLSTATUS"(ydh,khbh,khdh,kdsj)
values('B03642584120','312514','655254144444','2019-12-03 22:00:26');
SELECT * FROM "ZJS_ORDER"."ALADIN_WAYBILLSTATUS";
我们只看到了一条数据，这里并不奇怪，因为完成好整套索引之后，我们就是插入了一条数据。那么，之前的数据能否被索引呢？答案是肯定的。
3.2.13手动同步索引hadoop --config /etc/hadoop/conf \
jar /opt/cloudera/parcels/CDH/lib/hbase-solr/tools/hbase-indexer-mr-job.jar \
--conf /etc/hbase/conf/hbase-site.xml \
--hbase-indexer-file /opt/cdhsolr/zjs_order_aladin_waybillstatus/morphline-hbase-mapper.xml \
--morphline-file /opt/cdhsolr/zjs_order_aladin_waybillstatus/morphlines.conf \
--zk-host hbase1:2181,hbase2:2181,hbase3:2181/solr \
--collection zjs_order_aladin_waybillstatus \
--reducers 0 \
--go-live
上面的手动同步语句执行成功之后，我们再来观察solr中的索引数据
接下来，我们再手动插入一条数据
至此，一整套成熟的solr索引方案测试完毕。
4.删除索引假如要删除上面已创建的索引，依次执行如下命令即可，注意替换集合或索引名称
hbase-indexer delete-indexer --name zjs_order_aladin_waybillstatusIndexer
hbase-indexer list-indexers
solrctl collection --delete zjs_order_aladin_waybillstatus
solrctl instancedir --delete zjs_order_aladin_waybillstatus
作者：古佛青灯
