Skip to content

Commit d81a0ca

Browse files
bshafferbshaffer
authored
[BigQuery] Add Export Command and clean up Doc Blocks (GoogleCloudPlatform#144)
* BigQuery Samples Updates - moves code samples into separate functions - adds BrowseTableCommand, TablesCommand, and ProjectsCommand - adds GOOGLE_KEY_FILE for listing projects - moves README command docs to Command Help, removes failing permissions test
1 parent 31c94ae commit d81a0ca

34 files changed

+1863
-672
lines changed

‎bigquery/api/README.md

Lines changed: 9 additions & 112 deletions
Original file line numberDiff line numberDiff line change
@@ -23,118 +23,15 @@ This simple command-line application demonstrates how to invoke Google BigQuery
2323
5. Run `php bigquery.php`. The following commands are available:
2424

2525
```sh
26-
datasets List BigQuery datasets for a project
27-
import Import data into a BigQuery table
28-
query Run a BigQuery query
29-
schema Create or delete a table schema in BigQuery
30-
```
31-
32-
## The commands
33-
34-
### datasets
35-
36-
List the datasets for your BigQuery project.
37-
38-
```sh
39-
$ php bigquery.php datasets
40-
test_dataset1
41-
test_dataset2
42-
test_dataset3
43-
```
44-
45-
### import
46-
47-
Import data into a BigQuery table. You can import from several sources.
48-
49-
1. Import from a local JSON or CSV file. Make sure your files are
50-
[formatted correctly](https://cloud.google.com/bigquery/loading-data#specifying_the_source_format)
51-
52-
```sh
53-
$ php bigquery.php import test_dataset.test_table /path/to/your_data.csv
54-
$ php bigquery.php import test_dataset.test_table /path/to/your_data.json
55-
```
56-
1. Import from [a JSON or CSV file in Google Cloud Storage](https://cloud.google.com/bigquery/docs/loading-data-cloud-storage)
57-
58-
```sh
59-
$ php bigquery.php import test_dataset.test_table gs://your-storage-bucket/your_data.csv
60-
$ php bigquery.php import test_dataset.test_table gs://your-storage-bucket/your_data.json
61-
```
62-
1. Import from a [Datastore Backup](https://cloud.google.com/bigquery/loading-data-cloud-datastore)
63-
64-
```sh
65-
$ php bigquery.php import test_dataset.test_table gs://your-storage-bucket/your_data.backup_info
66-
```
67-
68-
You can also [stream data into bigquery](https://cloud.google.com/bigquery/streaming-data-into-bigquery)
69-
one record at a time. This approach enables querying data without the delay of running a load job:
70-
71-
```sh
72-
$ php bigquery.php import test_dataset.test_table
73-
Import data for project cloud-samples-tests-php? [y/n]: y
74-
name (required): Brent Shaffer
75-
title (required): PHP Developer
76-
Data streamed into BigQuery successfully
77-
```
78-
79-
### query
80-
81-
Run a BigQuery query
82-
83-
```sh
84-
$ php bigquery.php query "SELECT TOP(corpus, 3) as title, COUNT(*) as unique_words FROM [publicdata:samples.shakespeare]"
85-
--- Row 1 ---
86-
title: hamlet
87-
unique_words: 5318
88-
--- Row 2 ---
89-
title: kinghenryv
90-
unique_words: 5104
91-
--- Row 3 ---
92-
title: cymbeline
93-
unique_words: 4875
94-
```
95-
96-
### schema
97-
98-
Create a table schema in BigQuery. If a schema file is not supplied, you can
99-
create a schema interactively.
100-
101-
```sh
102-
$ php bigquery.php schema my_dataset.my_table --project your-project-id
103-
Using project your-project-id
104-
1st column name: name
105-
1st column type (default: string):
106-
[0] string
107-
[1] bytes
108-
[2] integer
109-
[3] float
110-
[4] boolean
111-
[5] timestamp
112-
[6] date
113-
[7] record
114-
> 0
115-
1st column mode (default: nullable):
116-
[0] nullable
117-
[1] required
118-
[2] repeated
119-
> 1
120-
add another field? [y/n]: n
121-
[
122-
{
123-
"name": "name",
124-
"type": "string",
125-
"mode": "required"
126-
}
127-
]
128-
Does this schema look correct? [y/n]: y
129-
Table created successfully
130-
```
131-
132-
The schema command also allows the deletion of tables
133-
```sh
134-
$ php bigquery.php schema my_dataset.my_table --project your-project-id --delete
135-
Using project your-project-id
136-
Are you sure you want to delete the BigQuery table "my_table"? [y/n]: y
137-
Table deleted successfully
26+
browse-table Browse a BigQuery table
27+
datasets List BigQuery datasets
28+
export Export data from a BigQuery table into a Cloud Storage bucket
29+
import Import data into a BigQuery table
30+
projects List BigQuery projects
31+
query Run a BigQuery query
32+
schema Create or delete a table schema in BigQuery
33+
tables List BigQuery tables
34+
6. Run `php bigquery.php COMMAND --help` to print information about the usage of each command.
13835
```
13936

14037
## Contributing changes

‎bigquery/api/bigquery.php

Lines changed: 12 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -2,19 +2,23 @@
22

33
require __DIR__ . '/vendor/autoload.php';
44

5+
use Google\Cloud\Samples\BigQuery\BrowseTableCommand;
56
use Google\Cloud\Samples\BigQuery\DatasetsCommand;
7+
use Google\Cloud\Samples\BigQuery\ExportCommand;
68
use Google\Cloud\Samples\BigQuery\ImportCommand;
9+
use Google\Cloud\Samples\BigQuery\ProjectsCommand;
710
use Google\Cloud\Samples\BigQuery\QueryCommand;
811
use Google\Cloud\Samples\BigQuery\SchemaCommand;
12+
use Google\Cloud\Samples\BigQuery\TablesCommand;
913
use Symfony\Component\Console\Application;
1014

11-
$datasets = new DatasetsCommand();
12-
$import = new ImportCommand();
13-
$query = new QueryCommand();
14-
$schema = new SchemaCommand();
1515
$application = new Application();
16-
$application->add($datasets);
17-
$application->add($import);
18-
$application->add($query);
19-
$application->add($schema);
16+
$application->add(new BrowseTableCommand());
17+
$application->add(new DatasetsCommand());
18+
$application->add(new ExportCommand());
19+
$application->add(new ImportCommand());
20+
$application->add(new ProjectsCommand());
21+
$application->add(new QueryCommand());
22+
$application->add(new SchemaCommand());
23+
$application->add(new TablesCommand());
2024
$application->run();

‎bigquery/api/composer.json

Lines changed: 11 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -9,7 +9,17 @@
99
"Google\\Cloud\\Samples\\BigQuery\\": "src/"
1010
},
1111
"files": [
12-
"src/functions.php"
12+
"src/functions/browse_table.php",
13+
"src/functions/delete_table.php",
14+
"src/functions/export_table.php",
15+
"src/functions/import_from_file.php",
16+
"src/functions/import_from_storage.php",
17+
"src/functions/list_datasets.php",
18+
"src/functions/list_projects.php",
19+
"src/functions/list_tables.php",
20+
"src/functions/run_query.php",
21+
"src/functions/run_query_as_job.php",
22+
"src/functions/stream_row.php"
1323
]
1424
}
1525
}

‎bigquery/api/composer.lock

Lines changed: 6 additions & 6 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎bigquery/api/src/BrowseTableCommand.php

Lines changed: 103 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,103 @@
1+
2+
/**
3+
* Copyright 2015 Google Inc. All Rights Reserved.
4+
*
5+
* Licensed under the Apache License, Version 2.0 (the "License");
6+
* you may not use this file except in compliance with the License.
7+
* You may obtain a copy of the License at
8+
*
9+
* http://www.apache.org/licenses/LICENSE-2.0
10+
*
11+
* Unless required by applicable law or agreed to in writing, software
12+
* distributed under the License is distributed on an "AS IS" BASIS,
13+
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
14+
* See the License for the specific language governing permissions and
15+
* limitations under the License.
16+
*/
17+
18+
namespace Google\Cloud\Samples\BigQuery;
19+
20+
use Google\Cloud\ClientTrait;
21+
use Symfony\Component\Console\Command\Command;
22+
use Symfony\Component\Console\Input\InputArgument;
23+
use Symfony\Component\Console\Input\InputInterface;
24+
use Symfony\Component\Console\Input\InputOption;
25+
use Symfony\Component\Console\Question\ConfirmationQuestion;
26+
use Symfony\Component\Console\Output\OutputInterface;
27+
use InvalidArgumentException;
28+
use Exception;
29+
30+
/**
31+
* Command line utility to list BigQuery tables.
32+
*
33+
* Usage: php bigquery.php tables
34+
*/
35+
class BrowseTableCommand extends Command
36+
{
37+
use ClientTrait;
38+
39+
protected function configure()
40+
{
41+
$this
42+
->setName('browse-table')
43+
->setDescription('Browse a BigQuery table')
44+
->setHelp(<<
45+
The %command.name% command outputs the rows of a BigQuery table.
46+
47+
php %command.full_name% DATASET.TABLE
48+
49+
EOF
50+
)
51+
->addArgument(
52+
'dataset.table',
53+
InputArgument::REQUIRED,
54+
'The dataset to list tables'
55+
)
56+
->addOption(
57+
'project',
58+
null,
59+
InputOption::VALUE_REQUIRED,
60+
'The Google Cloud Platform project name to use for this invocation. ' .
61+
'If omitted then the current gcloud project is assumed. '
62+
)
63+
->addOption(
64+
'max-results',
65+
null,
66+
InputOption::VALUE_REQUIRED,
67+
'The number of rows to return on each API call.',
68+
10
69+
)
70+
;
71+
}
72+
73+
protected function execute(InputInterface $input, OutputInterface $output)
74+
{
75+
if (!$projectId = $input->getOption('project')) {
76+
if (!$projectId = $this->detectProjectId()) {
77+
throw new Exception('Could not derive a project ID from gcloud. ' .
78+
'You must supply a project ID using --project');
79+
}
80+
}
81+
$maxResults = $input->getOption('max-results');
82+
$fullTableName = $input->getArgument('dataset.table');
83+
if (1 !== substr_count($fullTableName, '.')) {
84+
throw new InvalidArgumentException('Table must in the format "dataset.table"');
85+
}
86+
list($datasetId, $tableId) = explode('.', $fullTableName);
87+
88+
// create the function to determine if we should paginate
89+
$question = $this->getHelper('question');
90+
$q = new ConfirmationQuestion('[Press enter for next page, "n" to exit]');
91+
$shouldPaginate = function () use ($input, $output, $question, $q) {
92+
if (!$input->isInteractive()) {
93+
return false;
94+
}
95+
96+
return $question->ask($input, $output, $q);
97+
};
98+
99+
$totalRows = paginate_table($projectId, $datasetId, $tableId, $maxResults, $shouldPaginate);
100+
101+
printf('Found %s row(s)' . PHP_EOL, $totalRows);
102+
}
103+
}

‎bigquery/api/src/DatasetsCommand.php

Lines changed: 10 additions & 23 deletions
Original file line numberDiff line numberDiff line change
@@ -17,22 +17,27 @@
1717

1818
namespace Google\Cloud\Samples\BigQuery;
1919

20+
use Google\Cloud\ClientTrait;
2021
use Symfony\Component\Console\Command\Command;
2122
use Symfony\Component\Console\Input\InputInterface;
2223
use Symfony\Component\Console\Input\InputOption;
2324
use Symfony\Component\Console\Output\OutputInterface;
24-
use Google\Cloud\BigQuery\BigQueryClient;
25+
use Exception;
2526

2627
/**
2728
* Command line utility to list BigQuery datasets.
29+
*
30+
* Usage: php bigquery.php datasets
2831
*/
2932
class DatasetsCommand extends Command
3033
{
34+
use ClientTrait;
35+
3136
protected function configure()
3237
{
3338
$this
3439
->setName('datasets')
35-
->setDescription('List BigQuery datasets for a project')
40+
->setDescription('List BigQuery datasets')
3641
->setHelp(<<
3742
The %command.name% command lists all the datasets associated with your project.
3843
@@ -53,29 +58,11 @@ protected function configure()
5358
protected function execute(InputInterface $input, OutputInterface $output)
5459
{
5560
if (!$projectId = $input->getOption('project')) {
56-
if (!$projectId = $this->getProjectIdFromGcloud()) {
57-
throw new \Exception('Could not derive a project ID from gloud. ' .
61+
if (!$projectId = $this->detectProjectId()) {
62+
throw new Exception('Could not derive a project ID from gcloud. ' .
5863
'You must supply a project ID using --project');
5964
}
6065
}
61-
$bigQuery = new BigQueryClient([
62-
'projectId' => $projectId
63-
]);
64-
65-
# [START list_datasets]
66-
$datasets = $bigQuery->datasets();
67-
foreach ($datasets as $dataset) {
68-
$output->writeln(sprintf('%s', $dataset->id()));
69-
}
70-
# [END list_datasets]
71-
}
72-
73-
private function getProjectIdFromGcloud()
74-
{
75-
exec("gcloud config list --format 'value(core.project)' 2>/dev/null", $output, $return_var);
76-
77-
if (0 === $return_var) {
78-
return array_pop($output);
79-
}
66+
list_datasets($projectId);
8067
}
8168
}

0 commit comments

Comments
 (0)