How to manage several SSH Config files on Mac and Linux (quick-note)

Use the Include directive to include an external config file in the main (default) ssh config file.

  1. Create another config file as usual.
  2. Add the following line at the beginning of your ~/.ssh/config
    for example, if we have a config file in ~/Documents/.ssh/config add Include ~/Documents/.ssh/config in your main ~/.ssh/config

What is this good for?

As I mentioned in How to use multiple SSH keys on a Mac with GitHub or GitLab my objective is to s separate personal projects from work projects.

In this case, I can use an ssh config file for work and another file for my projects, and then include my projects into the main config file.

This also allows me to track my changes with a private git repository without worrying about confidential information from work.

References and further reading

How to use multiple SSH keys on a Mac with Github or Gitlab

A quick note on how to use multiple SSH Keys (Identities) on one machine.

A short note on the use case

Why would you want to use multiple keys on a machine you ask? There are several reasons, one being perceived security and another being convenience. Using just one Identity makes everything easier and you don’t have to manage several Identities You can read more here and here.

My use case would be that of convenience. I just want to separate work from personal projects and I would like to use a separate key (nonwork related) with those projects

1. Create a new public key

ssh-keygen -t rsa -C "[email protected]"

This public key can be in any path, not just in ~/.ssh/ as default/ recommended.

For security, it is recommended that you assign a unique password to this key.

Add all the keys to your cached keys

ssh-add path-to-key

For example, if you have your new key in ~/Documents/ssh_keys/new_key you would need to run ssh-add ~/Documents/ssh_keys/new_key. If you assigned a password in Setup 1, you will be asked for it.

Step 3: Update/ Create your ssh config

nano ~/.ssh/config

If you already have entries in your ssh config, just add a new entry at the end of the file. If your file is empty, or you didn’t have any before, just add the following.

Host gitlab-diego
    HostName gitlab.com # or github.com or any other domain
    User your-username # usually the one you use with that domain
    IdentityFile path-to-key

With the example data for GitLab and user diego this entry would look like the following:

Host gitlab-diego
    HostName gitlab.com
    User diego
    IdentityFile ~/Documents/ssh_keys/new_key

To save and close just press Ctrl + o and enter

To exit nano press Ctrl + x.

Step 4: Add the key to GitLab, GitHub or other services

In my case, I wanted to use the new key with GitLab. They have great instructions on how to do that but, for convenience, I list the steps here:

  1. Copy your new key to the clipboard.

  2. Log in into gitlab.com and go to https://gitlab.com/profile/keys

  3. Paste the key into the text box and click on „Add key“

Now you may clone a repository using this new key.

References and further reading

How to find files using Regex (Regular Expressions) in GNU/Linux and MacOs command line and do something.

I needed to find all the files that matched a regular expression. In this case all the files that had either 300x200 or 400x220 and where either png or jpg files.

You can do that pretty easily in BASH using find but the more file types and conditions the longer the command.

For example the following would be used to find all PNG and JPG files in /tmp folder:

find /tmp -name '*.png' -or -name '*.jpg'

Enter regular expressions :)

Important : the find command in macOS and in GNU/Linux (Ubuntu, Debian, etc..) are slightly different and not all syntax can be used between systems.

It’s a really useful magic but really hard to learn.

What is a regular expression?

A regular expression is a special text string for describing a search pattern. You can think of regular expressions as wildcards on steroids. You are probably familiar with wildcard notations such as *.txt to find all text files in a file manager. The regex equivalent is .*\.txt. -regexbuddy

On Mac

find -E . -regex '.*\.(jpg|png)'

On Linux

find ./ -regextype posix-extended -regex '.*(jpg|png)$'

Find all the jpg and png files which have 300x200 or 400x220 in their

filenames.

On Mac

find -E . -regex '.*(300x200|400x220)\.(jpg|png)'

On Linux

find ./ -regextype posix-extended -regex '.*(300x200|400x220)\.(jpg|png)$'

Those commands will find all the files of the type something-300x200.jpg or somethingelse400x220.png

So now you see a pattern. You can use the same regular expression on macOS and Linux (the part between ') if you change the syntax and add a $ at the end in the case of Linux.

Here are some useful regular expression you may want to know.

Find all the png and jpg files with SAM somewhere in the filename

'.*(SAM).*\.(jpg|png)'

This would find, among others, the following files


result of running the find command with '. (SAM)..(jpg|png)'

On Mac

find -E . -regex '.*(SAM).*\.(jpg|png)'

On Linux

find ./ -regextype posix-extended -regex '.*(SAM).*\.(jpg|png)$'

Now let’s learn xargs

xargs is a command on Unix and most Unix-like operating systems used to build and execute commands from standard input. It converts input from standard input into arguments to a command. -Wikipedia

It’s great to be able to find all those files, but you should be able to do something with them once you have found them, right?

That’s where xargs comes in. Let’s say you would like to remove all the JPG and PNG files that have SAM in the filename.

On Mac

find -E . -regex '.*(SAM).*\.(jpg|png)' | xargs rm

On Linux

find ./ -regextype posix-extended -regex '.*(SAM).*\.(jpg|png)$' | xargs rm

That was easy, right? And it’s really handy to be able to delete some files from all folders. For example all the automatically-made thumbnails on Wordpress and other systems. Let’s say you have to clean a folder with several sub-folders and only leave the original files. On a Mac you can do this (add or remove other resolutions) and remove all the files with a resolution in their name (for example something-AAAxBBB.jpg

find -E . -regex '.*(1024x576|278x380|506x380|1024x576|508x380|1024x681|550x366|550x309|350x220|380x380|768x768|285x380|768x1024|1024x1024|213x380|576x1024|550x367|253x380|681x1024|1024x768)\.(jpg|png)' | xargs rm

How to access Jupyter Notebooks running in your local server with ngrok (and an intro to GNU Screen)

I wrote about how to create a SSH Tunnel with ngrok on Ubuntu Server (aka ssh access to your local server in local network with dynamic ip) in another article. If you haven’t installed ngrok yet, please read that guide first.

After I had access to my Ubuntu Server through SSH, the next logical step (at least for me) was to enable Jupyter Notebooks running inside TORUS (Docker) to be reachable from a remote browser. That way I’m able to run python code on the server while using my browser as if it was locally installed. To run ngrok for a http port forward is fast and easy, even remotely through SSH, but before that let’s talk about screen

What is screen?

GNU Screen is a terminal multiplexer, a software application that can be used to multiplex several virtual consoles, allowing a user to access multiple separate login sessions inside a single terminal window, or detach and reattach sessions from a terminal. -Wikipedia

It is one of those tools you will love once you know them. screen lets you “open multiple windows” in your terminal (SSH included). You can run multiple commands and, if connected through SSH, keep running and get back to them even if the connection gets down. To install screen just open a terminal and type

sudo apt-get install screen -y

ngrok and screen

Once you have screen installed lets use ngrok to open a tunnel to your Jupyter instance inside TORUS.

1. SSH into your server 2. Open screen and assign a name

screen -S ngrok_jupyter`

3. Identify on which port is Jupyter Notebooks running

docker ps

4. Start ngrok on that port

ngrok http 32768

This will open ngrok with information on that connection, and you will see the following on your screen. If you access the URL you will open Jupyter Notebooks from anywhere.

5. Now lets use screen

Now you can press crtl + a and then d to leave (detach) screen.

ngrok will still be running. If your SSH connection goes down, ngrok will still be running.

6. To reconnect to (attach) screen and stop ngrok type

screen -r ngrok_jupyter

and press ctrl + c. This will interrupt ngrok

You should also know that

  • If the server connection gets down, you won’t be able to access it unless you are in front of it (physically).
  • You can use the same steps to forward any port from your local server to a public ngrok url. For example, a web server (apache2 or nginx), a custom app, a python server and so on. You only need to know the port on which you can access it locally and then write in when running ngrok

    ngrok http PORT

  • If you don’t want to use screen, you can append & at the end of the command to run it in the background.

  • If you don’t use screen nor append & your ngrok forward will stop if your SSH connection goes down.
  • You can also run ngrok locally. No need to use SSH if you are in front of the server.

Back to Windows 7 & 10 Home (for some tasks): chocolatey (package manager) and cmder (terminal emulator)

More than 10 years ago I switched to a Mac and never looked back (I still use Debian and/or Ubuntu for servers though, apt is the best!)… until now. For several reasons I’ve been dealing with windows for the last month on a daily basis and I’ve to say I miss the terminal so much! In MacOS I use iTerm2 for terminal emulator and brew and brew cask for app install and I love it and going back to Windows CMD and the whole click-click-click install is not what I would like. And I miss linux- like commands so much! Fortunately I found cmder and it’s working like a charm (it even has support for common unix-like commands like ls , cat and nano . That plus chocolatey is making my life so much easier! Think of chocolatey like brew for Mac or apt for Debian-based linux. Is so much better to search an app in a repository and install it from the command line than to open the browser, search for the website, find the download link, download the file, open the file, click all the buttons and repeat for each app you want. With cmder and chocolatey you have to (once both are installed):

  • To search for an app

    choco search name_of_app

  • To install an app

    choco install name_of_app

How to install cmder and chocolatey

To get both tools working just follow this steps:

  1. Open an administrator cmd and paste the following to install chocolatey

    bash @"%SystemRoot%\System32\WindowsPowerShell\v1.0\powershell.exe" -NoProfile -InputFormat None -ExecutionPolicy Bypass -Command "iex ((New-Object System.Net.WebClient).DownloadString('https://chocolatey.org/install.ps1'))" && SET "PATH=%PATH%;%ALLUSERSPROFILE%\chocolatey\bin" refreshenv

  2. Run the following and wait a few minutes

    choco install cmder -y

  3. Open cmder from the start menu and enjoy your new setup, with a cool terminal emulator and a repository of apps right at your finger tips.

About chocolatey: The package manager for Windows

Visit their website at https://chocolatey.org/

  • 4,737 Known good packages (apps; as of 2019-03-02)
  • Package submissions go through a rigorous moderation review process, including automatic virus scanning.
  • The open source version is enough for me. You can check their editions here

I’ll be using chocolatey in future how-to guides to install apps so stay tuned.

How to create a SSH Tunnel with ngrok on Ubuntu Server (aka ssh access to your local server in local network with dynamic ip)

I have a server at home running Ubuntu 18.04 LTS (normal local network, behind a router with firewall activated, no DMZ) and I wanted to be able to access it though ssh. You can think of it as a TeamViewer alternative for port redirection.

What’s SSH?

Secure Shell (SSH) is a cryptographic network protocol for operating network services securely over an unsecured network. Typical applications include remote command-line login and remote command execution, but any network service can be secured with SSH. -Wikipedia

Enter grok

Public URLs for exposing your local web server. Spend more time programming. One command for an instant, secure URL to your localhost server through any NAT or firewall.

They weren’t kidding. To set it up do the following:

  1. Register at https://ngrok.com/
  2. Follow the steps on their “Getting started page”. There are 4 steps to follow. If you are in Ubuntu just run the following in the terminal to install. Both methods work just fine.

    sudo snap install ngrok

And then specify your auth token form here

ngrok authtoken your-token

3. Start ngrok on your ssh port. You can read more here

./ngrok tcp 22

4. Access your server with ssh

ssh  YOUR_USER@0.tcp.ngrok.io -p PORT

Where PORT is the port assigned by ngrok and YOUR_USER is your user in the server. You can also check the port assigned at your ngrok dashboard

Optional: copy your ssh key to your server for a password-less access

I use ssh-copy-id as follows

ssh-copy-id -p port -v 0.tcp.eu.ngrok.io -l user

You can read more about it here On Ubuntu ssh-copy-id is already installed. On Mac just run

brew install ssh-copy-id

For more details about brew, read here. In a few words, think of brew as a package manager for Mac (like snap, apt, rpm, etc…). If you just want to get it installed open a terminal window and run

/usr/bin/ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"

About ngrok

ngrok is free for up to 10 devices. Their plans are in their pricing page

Their free plan allows the following:

  • HTTP/TCP tunnels on random URLs/ports
  • 1 online ngrok process
  • 4 tunnels/ngrok process
  • 40 connections / minute which is enough for personal and demo uses.

Enjoy!

How to prepare Ubuntu for Data Science: Torus, Cookiecutter and Docker in Ubuntu Server LTS 18.04


Important : since 2019-12 Tors is deprecated as stated in their GitHub Repository:


This project is deprecated and has been archived. A new, significantly improved, cookiecutter for Dockerized machine learning and data science can be found here: https://github.com/manifoldai/orbyter-cookiecutter.

I leave the original post just as a reference. ---

Original post:

I needed a faster, always-on computer and more storage and got a no-so-old (okay, it's old but has 4 cores and a 2 TB HD) HP Proliant ML110 with Ubuntu Linux LTS 18.04. I installed everything on my Mac, and doing it all over again made no sense. Not only that, but I also tried Anaconda (also a great tool to start, but you still have to install several packages on every new install). Then I found Torus: A Toolkit For Docker-First Data Science which you can download here or just read further.

Torus and Cookiecutter

Using the project cookiecutter and Docker image together you can go from cold-steel to a new project working in a Jupyter notebook with all the common libraries available to you in less than five minutes (and you didn’t have to pip install anything) -Alex Ng

Cookiecutter Data Science

A logical, reasonably standardized, but flexible project structure for doing and sharing data science work. -from its repository

What is Torus?

Torus is based on Cookiecutter Data Science project.

Helping Data Science teams easily move to a Docker-first development workflow to iterate and deliver projects faster and more reliably. The Torus 1.0 package contains a Python cookiecutter template and a public Docker image. The goal of Torus is to help data science teams adopt Docker and apply Development Operations (DevOps) best practices to streamline machine learning delivery pipelines. -From their website

In other words, install Torus and everything will be already working for you (pandas, Jupyter Notebook, sklearn, matplotlib, etc…). Should you want any package not included, you can always install it. The docker image provided is running Ubuntu, so there’s plenty of information out there. You can also install packages directly from Jupiter Notebooks using pip, which is recommended to have reproducible code. Also, you will find the data (files and folders) in data in your project’s folder. The docker image mounts this folder as a local folder inside Torus as /mnt, and thus you can use your normal tools (for example Sublime Text on my Mac) to edit files with are later run inside Torus. You can also access Jupiter Notebooks from your browser.

How to install Torus and Docker

These are the steps to prepare the system for Data Science stuffs by installing Torus. Open your terminal and follow these steps: 1. Download Docker CE and Python PIP (this assumes you already have python installed. That’s the default btw.).

    sudo curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -sudo add-apt-repository \   "deb [arch=amd64] https://download.docker.com/linux/ubuntu \   $(lsb_release -cs) \   stable"sudo apt-get install docker-ce docker-ce-cli containerd.io python-pip
  1. Install coockiecutter
  sudo pip install cookiecutter
  1. Give permissions to your user (taken from here)
    sudo usermod -a -G docker $USER
  1. Install docker-composer (taken from here)
    sudo curl -L "https://github.com/docker/compose/releases/download/1.23.2/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose
  1. Run the following in the folder of your choice (you can change the folder if you want
    mkdir Docker-DataScience && cd Docker-DataSciencecookiecutter https://github.com/manifoldai/docker-cookiecutter-data-science.git
  1. Enter the name of the Project. You can just use the following if you don’t know
  2. Enter the following (change data-science-docker for the name you gave your project). This will download around 430mb.
    cd data-science-docker./start.sh

Things you should know about TORUS

    docker ps

  • Docker uses resources, so your computer should be slower and may heat (depends on the computer). If you are not using TORUS remember to stop the container.
    docker stop <name-of-the-container>

Tip: if you write “docker stop “ and the first letter of the name of the container and press tab, it will fill the rest with the name

  • You can start the container with
    docker start <name-of-the-container>

Remember to check in which port it’s available after starting. The port might change

  • Get the port on which Torus is running to access Jupyter Notebooks and get the name of the container.
    docker ps
  • Get a bash shell in the container. Replace
    docker exec -it <container name> /bin/bash
  • once in the bash shell, you can add a user (not a good idea to be always logged in as root
    adduser <new_user>
  • login as that new user
    su -l <new_user>

Data Science - A small intro, what enticed me and a few resources to start

Notes from IBM Data Science Professional Certificate course by IBM on Coursera__Notes by Diego Carrasco G.

I’m starting a small series of articles regarding Data Science, what enticed me, why I started, where to start and so on. I’m as of now by no way an expert but an enthusiastic learner who’s writing to learn better, share what I’ve learned and as an introduction and basis for others who would want to start in this area. Right now I’m on an IBM Course on Coursera which I find to be really great (and so far from all the courses I’ve tried to take on Coursera the IBM ones are the only ones I’ve finished, and quickly may I add.)

image-1

I finished 7 courses from the IBM Data Science Professional Certificate, and I'm on the 6th week of the 8th one. It was the best trial ever (7 days, 6 IBM/Coursera Certificates and 6 IBM Badges by youracclaim.com) Honestly, I couldn't recommend this set of courses enough. It's really quite a good basis to start in Data Science and Python. And it all began because I wanted to invest in Real Estate and wanted to have a Data Science base of all the freely available property ads in Chile using "www.yapo.cl" (its only accessible if you are in Chile, but a VPN works like a charm). I wrote a Python Script fast to scrap the data (15k rows up today for the last 6 months) using requests and bs4. That got me started, but Excel took quite a lot of time to render the data and I still had to clean it up a bit more. I had read about python, pandas and so on and searched how to use those tools, and I stumped right into Coursera and this Professional Certificate. I also looked in EDX, class-central and many others, but this one gave me a certificate (always good), the Data Analysis with Python was enjoyable and fast-paced, and I finished it in a couple of days, leaving me quite excited for more. I realized I like this area and started the other courses right away, and I’m now on the 8th course on Machine Learning with Python (it’s harder that the others, and it’s taking more time to finish it, but I’m excited about the possibilities and astonished as to how many tools and models are available. Pandas is also quite a useful discovery for me, and I’m loving it).

Handwriten notes

I’ve taken lots of notes (by hand, hence the picture above) and I’ll be re-writing them in this site. Maybe they are helpful for others. While I’m working on the Capstone Project I was in need of a way to get a DataFrame of venues near a coordinate using Foursquare API, so I developed a small package in python and published it to GitHub. You are free to check it out and play with it. Any recommendations are welcome, it’s my first package after all. And it’s under the MIT License, so enjoy. You can check that project on GitHub and check a small post on that project here

Python package to get venues from Foursquare API into Pandas DataFrame

Important: I do not use this anymore and is not being maintained. This post is here just as reference.

I’m working on my Capstone project for the IBM Data Science Professional Certificate Specialization on Coursera and wanted a fast and easy way to get venues around some coordinates. The way the explained in the course is fine, but I would like to have the data into a DataFrame ready to use.
I found https://pypi.python.org/pypi/foursquare/() which works fantastic, but still had to clean the result, thus this.
I developed a small package called foursquare_api_tools that returns the venues in a dataframe with several columns. You can check the package in GitHub

You can use it in Jupyter Notebooks without problems. I tested it in IBM Cloud and https://labs.cognitiveclass.ai
Just install it using !pip install foursquare

Example of use

Import libraries

import foursquare as fs from foursquare_api_tools
import foursquare_api_tools as ft

Initialize the client

CLIENT_ID = 'YOUR_CLIENT_ID' # your Foursquare ID
CLIENT_SECRET = 'YOUR_SECRET' # your Foursquare Secret 
VERSION = '20180605' # Foursquare API version 

this example uses Userless Auth

# Construct the client object using foursquare package
client = fs.Foursquare(client_id=CLIENT_ID, client_secret=CLIENT_SECRET,
version=VERSION)

Use the function

venues_explore(client,lat='40.7233',lng='-74.0030',limit=100,
offset=2)

Example

Here is an example notebook created on dataplatform.cloud.ibm.com

{
    "nbformat_minor": 1, 
    "cells": [
        {
            "execution_count": 9, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [
                {
                    "output_type": "stream", 
                    "name": "stdout", 
                    "text": "Requirement not upgraded as not directly required: foursquare in /opt/conda/envs/DSX-Python35/lib/python3.5/site-packages\nRequirement not upgraded as not directly required: requests>=2.1 in /opt/conda/envs/DSX-Python35/lib/python3.5/site-packages (from foursquare)\nRequirement not upgraded as not directly required: six in /opt/conda/envs/DSX-Python35/lib/python3.5/site-packages (from foursquare)\nRequirement not upgraded as not directly required: chardet<3.1.0,>=3.0.2 in /opt/conda/envs/DSX-Python35/lib/python3.5/site-packages (from requests>=2.1->foursquare)\nRequirement not upgraded as not directly required: idna<2.7,>=2.5 in /opt/conda/envs/DSX-Python35/lib/python3.5/site-packages (from requests>=2.1->foursquare)\nRequirement not upgraded as not directly required: urllib3<1.23,>=1.21.1 in /opt/conda/envs/DSX-Python35/lib/python3.5/site-packages (from requests>=2.1->foursquare)\nRequirement not upgraded as not directly required: certifi>=2017.4.17 in /opt/conda/envs/DSX-Python35/lib/python3.5/site-packages (from requests>=2.1->foursquare)\nCollecting foursquare_api_tools from git+https://github.com/dacog/foursquare_api_tools.git#egg=foursquare_api_tools\n  Cloning https://github.com/dacog/foursquare_api_tools.git to /home/dsxuser/.tmp/pip-build-ehhz4mgs/foursquare-api-tools\nRequirement not upgraded as not directly required: pandas in /opt/conda/envs/DSX-Python35/lib/python3.5/site-packages (from foursquare_api_tools)\nRequirement not upgraded as not directly required: foursquare in /opt/conda/envs/DSX-Python35/lib/python3.5/site-packages (from foursquare_api_tools)\nRequirement not upgraded as not directly required: python-dateutil>=2 in /opt/conda/envs/DSX-Python35/lib/python3.5/site-packages (from pandas->foursquare_api_tools)\nRequirement not upgraded as not directly required: pytz>=2011k in /opt/conda/envs/DSX-Python35/lib/python3.5/site-packages (from pandas->foursquare_api_tools)\nRequirement not upgraded as not directly required: numpy>=1.9.0 in /opt/conda/envs/DSX-Python35/lib/python3.5/site-packages (from pandas->foursquare_api_tools)\nRequirement not upgraded as not directly required: requests>=2.1 in /opt/conda/envs/DSX-Python35/lib/python3.5/site-packages (from foursquare->foursquare_api_tools)\nRequirement not upgraded as not directly required: six in /opt/conda/envs/DSX-Python35/lib/python3.5/site-packages (from foursquare->foursquare_api_tools)\nRequirement not upgraded as not directly required: chardet<3.1.0,>=3.0.2 in /opt/conda/envs/DSX-Python35/lib/python3.5/site-packages (from requests>=2.1->foursquare->foursquare_api_tools)\nRequirement not upgraded as not directly required: idna<2.7,>=2.5 in /opt/conda/envs/DSX-Python35/lib/python3.5/site-packages (from requests>=2.1->foursquare->foursquare_api_tools)\nRequirement not upgraded as not directly required: urllib3<1.23,>=1.21.1 in /opt/conda/envs/DSX-Python35/lib/python3.5/site-packages (from requests>=2.1->foursquare->foursquare_api_tools)\nRequirement not upgraded as not directly required: certifi>=2017.4.17 in /opt/conda/envs/DSX-Python35/lib/python3.5/site-packages (from requests>=2.1->foursquare->foursquare_api_tools)\nInstalling collected packages: foursquare-api-tools\n  Found existing installation: foursquare-api-tools 0.1\n    Uninstalling foursquare-api-tools-0.1:\n      Successfully uninstalled foursquare-api-tools-0.1\n  Running setup.py install for foursquare-api-tools ... \u001b[?25ldone\n\u001b[?25hSuccessfully installed foursquare-api-tools-0.1\n"
                }
            ], 
            "source": "!pip install foursquare\n!pip install git+https://github.com/dacog/foursquare_api_tools.git#egg=foursquare_api_tools --upgrade --force-reinstall\n\nimport foursquare as fs\n"
        }, 
        {
            "execution_count": 11, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [], 
            "source": "# The code was removed by Watson Studio for sharing."
        }, 
        {
            "execution_count": 12, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [], 
            "source": "# Construct the client object \nclient = fs.Foursquare(client_id=CLIENT_ID, client_secret=CLIENT_SECRET, version=VERSION)"
        }, 
        {
            "execution_count": 13, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [
                {
                    "execution_count": 13, 
                    "metadata": {}, 
                    "data": {
                        "text/html": "
\n\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n
indexAddressCategoryCityCountryLatitudeLongitudeNamePostal Code
00459 Broome StArts & Crafts StoreNew YorkUnited States40.722326-74.000994Purl Soho10013
1042 Wooster StSalon / BarbershopNew YorkUnited States40.722371-74.002562Sam Brocato Salon10013
20466 Broome StDance StudioNew YorkUnited States40.722578-74.001363Dance With Me SoHo10013
30484 Broome StDessert ShopNew YorkUnited States40.723101-74.002477MarieBelle10013
40171 Spring StTapas RestaurantNew YorkUnited States40.724800-74.002220Boqueria10012
5071 Greene StMen's StoreNew YorkUnited States40.723427-74.000742ONS Clothing10012
60494 Broome StSupermarketNew YorkUnited States40.723192-74.002775Sunrise Mart10013
70148 Spring StShoe StoreNew YorkUnited States40.724261-74.001328Dr. Martens10012
80469 Broome StWomen's StoreNew YorkUnited States40.722630-74.001511Isabel Marant10013
9035 Wooster StArt MuseumNew YorkUnited States40.722393-74.002730The Drawing Center10013
100453 Broome StMen's StoreNew YorkUnited States40.722173-74.001111SuitSupply10013
110398 W BroadwayDessert ShopNew YorkUnited States40.724523-74.002748Ladur\u00e9e10012
12028-26 Wooster StArt GalleryNew YorkUnited States40.721780-74.003170Leslie+Lohman Museum of Gay & Lesbian Art10013
13090 Thompson StItalian RestaurantNew YorkUnited States40.725223-74.002594San Carlo Osteria Piemonte10012
140512 Broome StBakeryNew YorkUnited States40.723516-74.003444Pi Greek Bakerie10013
15076 Greene St Fl 3Design StudioNew YorkUnited States40.723466-74.000606The Line \u2013 New York10012
160189 Spring StBakeryNew YorkUnited States40.725163-74.002976Dominique Ansel Bakery10012
170529 Broome StBurger JointNew YorkUnited States40.723897-74.004221Black Tap10013
180470 Broome StArt GalleryNew YorkUnited States40.722607-74.001432Eden Fine Art10013
190510 Broome StItalian RestaurantNew YorkUnited States40.723426-74.003277Aurora10013
20093 Grand StShoe StoreNew YorkUnited States40.721534-74.002109The Vans DQM General10013
210456 Broome StBoutiqueNew YorkUnited States40.722279-74.000879Henrik Vibskov10013
22073 Thompson StJapanese RestaurantNew YorkUnited States40.724566-74.002873Hirohisa10012
230149 Spring StBoutiqueNew YorkUnited States40.724372-74.001496LF Boutique10012
240399 W BroadwayCoffee ShopNew YorkUnited States40.724292-74.002120Ground Support10012
25071 Sullivan StFalafel RestaurantNew YorkUnited States40.724523-74.004088Ba'al Cafe10012
260139 Spring StBoutiqueNew YorkUnited States40.724289-74.000843CHANEL10012
27033 Greene StClothing StoreNew YorkUnited States40.721688-74.002298Acne Studios10013
280477 Broome StWomen's StoreNew YorkUnited States40.722684-74.002160Kirna Zab\u00eate10013
29015 Thompson St.Cocktail BarNew YorkUnited States40.722671-74.004812JIMMY at The James10013
..............................
70015 Mercer StClothing StoreNew YorkUnited States40.720585-74.0020773x110013
7101 York StGymNew YorkUnited States40.721128-74.005398Barry's Bootcamp TriBeCa10013
720Wooster StMexican RestaurantNew YorkUnited States40.725482-73.999950Calexico Cart10012
730160 Prince StBakeryNew YorkUnited States40.725968-74.001254Birdbath Neighborhood Green Bakery10012
740486 BroadwayWomen's StoreNew YorkUnited States40.721557-73.999712Madewell10013
750451 BroadwayFurniture / Home StoreNew YorkUnited States40.720590-74.001060CB210013
760190 Avenue of the AmericasItalian RestaurantNew YorkUnited States40.726395-74.003696Bar Ciccio Alimentari10013
770433 Broome StTea RoomNew YorkUnited States40.721451-73.999394Harney & Sons10013
780512 BroadwayClothing StoreNew YorkUnited States40.722491-73.999071AllSaints10012
790131 Sullivan StNew American RestaurantNew YorkUnited States40.726542-74.002252The Dutch10012
800454 W BroadwayCosmetics ShopNew YorkUnited States40.726182-74.000795Benefit Cosmetics10012
810131 Mercer StClothing StoreNew YorkUnited States40.724413-73.998758A.P.C.10012
820121 Greene StOptical ShopNew YorkUnited States40.725390-73.999140Warby Parker10012
830101 Prince StWatch ShopNew YorkUnited States40.724898-73.999035Shinola Soho10012
84045 Crosby StCycle StudioNew YorkUnited States40.721899-73.998717SoulCycle SoHo10012
850319 Church StCoffee ShopNew YorkUnited States40.719829-74.003854La Colombe Torrefaction10013
86050 Varick StEvent SpaceNew YorkUnited States40.720915-74.006207Spring Studios10013
870463 W BroadwayBagel ShopNew YorkUnited States40.726316-74.000357Sadelle's10012
880311 Church StAsian RestaurantNew YorkUnited States40.719759-74.003981Macao Trading Co.10013
89032 Avenue of the AmericasTheaterNew YorkUnited States40.720054-74.004339The Drama League Theater Center10013
90099 Prince StAmerican RestaurantNew YorkUnited States40.724688-73.998675Mercer Kitchen10012
910142 Mercer StSeafood RestaurantNew YorkUnited States40.724572-73.998469Lure Fishbar10012
920127 Grand StVegetarian / Vegan RestaurantNew YorkUnited States40.720545-74.000138Le Botaniste10013
930103 Prince StreetElectronics StoreNew YorkUnited States40.725058-73.999029Apple SoHo10012
94080 Spring StFrench RestaurantNew YorkUnited States40.722724-73.998170Balthazar10012
950458 W BroadwaySalon / BarbershopNew YorkUnited States40.726214-74.000616Serenity Salon10012
960155 Varick StWine BarNew YorkUnited States40.726240-74.005773City Winery10013
97031 Crosby StClothing StoreNew YorkUnited States40.720746-73.999346Saturdays Surf NYC10013
980199 Prince StFrench RestaurantNew YorkUnited States40.726790-74.002750Little Prince10012
99038 Macdougal StMediterranean RestaurantNew YorkUnited States40.727083-74.002914Shuka10012
\n

100 rows \u00d7 9 columns

\n
", "text/plain": " index Address Category \\\n0 0 459 Broome St Arts & Crafts Store \n1 0 42 Wooster St Salon / Barbershop \n2 0 466 Broome St Dance Studio \n3 0 484 Broome St Dessert Shop \n4 0 171 Spring St Tapas Restaurant \n5 0 71 Greene St Men's Store \n6 0 494 Broome St Supermarket \n7 0 148 Spring St Shoe Store \n8 0 469 Broome St Women's Store \n9 0 35 Wooster St Art Museum \n10 0 453 Broome St Men's Store \n11 0 398 W Broadway Dessert Shop \n12 0 28-26 Wooster St Art Gallery \n13 0 90 Thompson St Italian Restaurant \n14 0 512 Broome St Bakery \n15 0 76 Greene St Fl 3 Design Studio \n16 0 189 Spring St Bakery \n17 0 529 Broome St Burger Joint \n18 0 470 Broome St Art Gallery \n19 0 510 Broome St Italian Restaurant \n20 0 93 Grand St Shoe Store \n21 0 456 Broome St Boutique \n22 0 73 Thompson St Japanese Restaurant \n23 0 149 Spring St Boutique \n24 0 399 W Broadway Coffee Shop \n25 0 71 Sullivan St Falafel Restaurant \n26 0 139 Spring St Boutique \n27 0 33 Greene St Clothing Store \n28 0 477 Broome St Women's Store \n29 0 15 Thompson St. Cocktail Bar \n.. ... ... ... \n70 0 15 Mercer St Clothing Store \n71 0 1 York St Gym \n72 0 Wooster St Mexican Restaurant \n73 0 160 Prince St Bakery \n74 0 486 Broadway Women's Store \n75 0 451 Broadway Furniture / Home Store \n76 0 190 Avenue of the Americas Italian Restaurant \n77 0 433 Broome St Tea Room \n78 0 512 Broadway Clothing Store \n79 0 131 Sullivan St New American Restaurant \n80 0 454 W Broadway Cosmetics Shop \n81 0 131 Mercer St Clothing Store \n82 0 121 Greene St Optical Shop \n83 0 101 Prince St Watch Shop \n84 0 45 Crosby St Cycle Studio \n85 0 319 Church St Coffee Shop \n86 0 50 Varick St Event Space \n87 0 463 W Broadway Bagel Shop \n88 0 311 Church St Asian Restaurant \n89 0 32 Avenue of the Americas Theater \n90 0 99 Prince St American Restaurant \n91 0 142 Mercer St Seafood Restaurant \n92 0 127 Grand St Vegetarian / Vegan Restaurant \n93 0 103 Prince Street Electronics Store \n94 0 80 Spring St French Restaurant \n95 0 458 W Broadway Salon / Barbershop \n96 0 155 Varick St Wine Bar \n97 0 31 Crosby St Clothing Store \n98 0 199 Prince St French Restaurant \n99 0 38 Macdougal St Mediterranean Restaurant \n\n City Country Latitude Longitude \\\n0 New York United States 40.722326 -74.000994 \n1 New York United States 40.722371 -74.002562 \n2 New York United States 40.722578 -74.001363 \n3 New York United States 40.723101 -74.002477 \n4 New York United States 40.724800 -74.002220 \n5 New York United States 40.723427 -74.000742 \n6 New York United States 40.723192 -74.002775 \n7 New York United States 40.724261 -74.001328 \n8 New York United States 40.722630 -74.001511 \n9 New York United States 40.722393 -74.002730 \n10 New York United States 40.722173 -74.001111 \n11 New York United States 40.724523 -74.002748 \n12 New York United States 40.721780 -74.003170 \n13 New York United States 40.725223 -74.002594 \n14 New York United States 40.723516 -74.003444 \n15 New York United States 40.723466 -74.000606 \n16 New York United States 40.725163 -74.002976 \n17 New York United States 40.723897 -74.004221 \n18 New York United States 40.722607 -74.001432 \n19 New York United States 40.723426 -74.003277 \n20 New York United States 40.721534 -74.002109 \n21 New York United States 40.722279 -74.000879 \n22 New York United States 40.724566 -74.002873 \n23 New York United States 40.724372 -74.001496 \n24 New York United States 40.724292 -74.002120 \n25 New York United States 40.724523 -74.004088 \n26 New York United States 40.724289 -74.000843 \n27 New York United States 40.721688 -74.002298 \n28 New York United States 40.722684 -74.002160 \n29 New York United States 40.722671 -74.004812 \n.. ... ... ... ... \n70 New York United States 40.720585 -74.002077 \n71 New York United States 40.721128 -74.005398 \n72 New York United States 40.725482 -73.999950 \n73 New York United States 40.725968 -74.001254 \n74 New York United States 40.721557 -73.999712 \n75 New York United States 40.720590 -74.001060 \n76 New York United States 40.726395 -74.003696 \n77 New York United States 40.721451 -73.999394 \n78 New York United States 40.722491 -73.999071 \n79 New York United States 40.726542 -74.002252 \n80 New York United States 40.726182 -74.000795 \n81 New York United States 40.724413 -73.998758 \n82 New York United States 40.725390 -73.999140 \n83 New York United States 40.724898 -73.999035 \n84 New York United States 40.721899 -73.998717 \n85 New York United States 40.719829 -74.003854 \n86 New York United States 40.720915 -74.006207 \n87 New York United States 40.726316 -74.000357 \n88 New York United States 40.719759 -74.003981 \n89 New York United States 40.720054 -74.004339 \n90 New York United States 40.724688 -73.998675 \n91 New York United States 40.724572 -73.998469 \n92 New York United States 40.720545 -74.000138 \n93 New York United States 40.725058 -73.999029 \n94 New York United States 40.722724 -73.998170 \n95 New York United States 40.726214 -74.000616 \n96 New York United States 40.726240 -74.005773 \n97 New York United States 40.720746 -73.999346 \n98 New York United States 40.726790 -74.002750 \n99 New York United States 40.727083 -74.002914 \n\n Name Postal Code \n0 Purl Soho 10013 \n1 Sam Brocato Salon 10013 \n2 Dance With Me SoHo 10013 \n3 MarieBelle 10013 \n4 Boqueria 10012 \n5 ONS Clothing 10012 \n6 Sunrise Mart 10013 \n7 Dr. Martens 10012 \n8 Isabel Marant 10013 \n9 The Drawing Center 10013 \n10 SuitSupply 10013 \n11 Ladur\u00e9e 10012 \n12 Leslie+Lohman Museum of Gay & Lesbian Art 10013 \n13 San Carlo Osteria Piemonte 10012 \n14 Pi Greek Bakerie 10013 \n15 The Line \u2013 New York 10012 \n16 Dominique Ansel Bakery 10012 \n17 Black Tap 10013 \n18 Eden Fine Art 10013 \n19 Aurora 10013 \n20 The Vans DQM General 10013 \n21 Henrik Vibskov 10013 \n22 Hirohisa 10012 \n23 LF Boutique 10012 \n24 Ground Support 10012 \n25 Ba'al Cafe 10012 \n26 CHANEL 10012 \n27 Acne Studios 10013 \n28 Kirna Zab\u00eate 10013 \n29 JIMMY at The James 10013 \n.. ... ... \n70 3x1 10013 \n71 Barry's Bootcamp TriBeCa 10013 \n72 Calexico Cart 10012 \n73 Birdbath Neighborhood Green Bakery 10012 \n74 Madewell 10013 \n75 CB2 10013 \n76 Bar Ciccio Alimentari 10013 \n77 Harney & Sons 10013 \n78 AllSaints 10012 \n79 The Dutch 10012 \n80 Benefit Cosmetics 10012 \n81 A.P.C. 10012 \n82 Warby Parker 10012 \n83 Shinola Soho 10012 \n84 SoulCycle SoHo 10012 \n85 La Colombe Torrefaction 10013 \n86 Spring Studios 10013 \n87 Sadelle's 10012 \n88 Macao Trading Co. 10013 \n89 The Drama League Theater Center 10013 \n90 Mercer Kitchen 10012 \n91 Lure Fishbar 10012 \n92 Le Botaniste 10013 \n93 Apple SoHo 10012 \n94 Balthazar 10012 \n95 Serenity Salon 10012 \n96 City Winery 10013 \n97 Saturdays Surf NYC 10013 \n98 Little Prince 10012 \n99 Shuka 10012 \n\n[100 rows x 9 columns]" }, "output_type": "execute_result" } ], "source": "from foursquare_api_tools import foursquare_api_tools as ft\n\n#import pandas as pd\n\nft.venues_explore(client,lat='40.7233',lng='-74.0030',limit=100)" }, { "execution_count": 14, "cell_type": "code", "metadata": {}, "outputs": [ { "output_type": "stream", "name": "stdout", "text": "Help on module foursquare_api_tools.foursquare_api_tools in foursquare_api_tools:\n\nNAME\n foursquare_api_tools.foursquare_api_tools\n\nFUNCTIONS\n venues_explore(client, lat, lng, limit)\n funtion to get n-places using explore in foursquare, where n is the limit when calling the function.\n This returns a pandas dataframe with name, city ,country, lat, long, postal code, address and main category as columns\n\nFILE\n /opt/conda/envs/DSX-Python35/lib/python3.5/site-packages/foursquare_api_tools/foursquare_api_tools.py\n\n\n" } ], "source": "help(ft)" }, { "execution_count": 15, "cell_type": "code", "metadata": {}, "outputs": [ { "execution_count": 15, "metadata": {}, "data": { "text/plain": "['__builtins__',\n '__cached__',\n '__doc__',\n '__file__',\n '__loader__',\n '__name__',\n '__package__',\n '__spec__',\n 'pd',\n 'venues_explore']" }, "output_type": "execute_result" } ], "source": "dir(ft)" }, { "execution_count": 16, "cell_type": "code", "metadata": {}, "outputs": [ { "output_type": "stream", "name": "stdout", "text": "/opt/conda/envs/DSX-Python35/bin/python\r\n" } ], "source": "!which python" }, { "execution_count": 17, "cell_type": "code", "metadata": {}, "outputs": [ { "output_type": "stream", "name": "stdout", "text": "python: /usr/bin/python2.7 /usr/bin/python /usr/lib/python2.7 /usr/lib64/python2.7 /etc/python /usr/include/python2.7 /opt/conda/bin/python3.6m /opt/conda/bin/python3.6-config /opt/conda/bin/python3.6m-config /opt/conda/bin/python3.6 /opt/conda/bin/python /opt/conda/envs/DSX-Python35/bin/python3.5 /opt/conda/envs/DSX-Python35/bin/python3.5m-config /opt/conda/envs/DSX-Python35/bin/python3.5-config /opt/conda/envs/DSX-Python35/bin/python3.5m /opt/conda/envs/DSX-Python35/bin/python\r\n" } ], "source": "!whereis python" }, { "execution_count": null, "cell_type": "code", "metadata": {}, "outputs": [], "source": "" } ], "metadata": { "kernelspec": { "display_name": "Python 3.5", "name": "python3", "language": "python" }, "language_info": { "mimetype": "text/x-python", "nbconvert_exporter": "python", "version": "3.5.5", "name": "python", "file_extension": ".py", "pygments_lexer": "ipython3", "codemirror_mode": { "version": 3, "name": "ipython" } } }, "nbformat": 4 }

Example output

Important

This is the first version, and I’m using it for the project, but it can still have some bugs.
I’ll add more information and details in the future. In the meanwhile you are welcome to check the code on GitHub.